Method and system for improving text-to-content recommendations using unsupervised learning
Through the combination of unsupervised and supervised machine learning models, pre-training embed training text into content suggestions system, the model training problems under user privacy and data anonymity requirements are solved, the accuracy and efficiency of suggestions are improved, and new content is realized timely presentation.
Patent Information
- Application Number
- CN201980044681.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-05-01
AI Technical Summary
Existing text-to-content suggestions systems are difficult to properly train the model when dealing with user privacy and data anonymity requirements, resulting in low quality suggestions, especially when new content is added, and content suggestions are not allowed due to unbalanced tagged data. precise.
Unsupervised machine learning model is used to pre-train embeddings for unlabeled data, and combined with supervised models to train text-to-content suggestions models, and use pre-trained embeddings to improve the recommendation accuracy and efficiency of the model.
提高了文本到内容模型的训练效率和建议准确性,确保新内容的及时呈现,提供了更用户友好的应用程序体验,提高了系统和用户效率。
Smart Images

Figure CN112384909B_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to methods, systems, and computer-readable media for improving text-to-content recommendations using unsupervised learning. Background Art
[0002] Intelligent text-to-content suggestion services are used in a wide variety of computer programs. For example, a text-to-content suggestion service can be used to suggest images, icons, or emoticons based on text received as input in an application. Typically, such applications may need to comply with certain privacy and data regulations. Therefore, these applications may not be able to store and use input user data. In fact, they may utilize mechanisms such as masking certain words and not maintaining the original sentence order of the input text to ensure privacy. This makes it difficult to correctly make text-to-content suggestions.
[0003] In addition, new content is often provided to applications that provide text-to-content suggestions. Because the process of labeling new content is often time-consuming and laborious, some of the new content may be unlabeled. Generally, the more labeled data a model contains, the better the quality of the suggestions provided by the process. For example, when adding new unlabeled content to a text-to-content service, the chance of providing the new data as a suggestion is lower than using old data. This may mean that even if new content is added to the service, it may not be presented to the user for a long time.
[0004] Therefore, there is a need for improved systems and methods for intelligently training text-to-content suggestion models. Summary of the invention
[0005] In one general aspect, the present disclosure provides a device having a processor and a memory in communication with the processor, wherein the memory stores executable instructions that, when executed by the processor, cause the device to perform multiple functions. The functions may include: accessing a dataset containing unlabeled training data collected from an application, the unlabeled training data collected under user privacy constraints; applying an unsupervised machine learning (ML) model to the dataset to generate pre-trained embeddings; applying a supervised ML model to a labeled dataset to train a text-to-content suggestion ML model utilized by the application. The supervised ML model may train the text-to-content suggestion ML model using the pre-trained embeddings generated by the supervised ML model.
[0006] In another general aspect, the present application describes a method for training a text-to-content suggestion ML model. The method may include: accessing a dataset containing unlabeled training data collected from an application, the unlabeled training data collected under user privacy constraints; applying an unsupervised machine learning (ML) model to the dataset to generate pre-trained embeddings; applying a supervised ML model to the labeled dataset to train a text-to-content suggestion ML model utilized by the application. The supervised ML model may utilize the pre-trained embeddings generated by the supervised ML model to train the text-to-content suggestion ML model.
[0007] In another general aspect, the present application describes a non-transitory computer-readable medium having instructions stored thereon, which, when executed, cause a programmable device to: access a dataset containing unlabeled training data collected from an application, the unlabeled training data being collected under user privacy constraints; apply an unsupervised machine learning (ML) model to the dataset to generate pre-trained embeddings; apply a supervised ML model to the labeled dataset to train a text-to-content suggestion ML model utilized by the application. The supervised ML model can train the text-to-content suggestion ML model using the pre-trained embeddings generated by the supervised ML model.
[0008] This summary is provided to introduce in simplified form a selection of concepts that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the present invention, nor is it intended to limit the scope of protection of the present invention. In addition, the claimed subject matter is not limited to embodiments that address any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The accompanying drawings depict one or more embodiments according to the present teachings by way of example only and not limitation. In the accompanying drawings, the same reference numerals represent the same or similar elements. In addition, it should be understood that the accompanying drawings are not necessarily drawn to scale.
[0010] Figure 1 An exemplary system is depicted upon which aspects of the present disclosure may be implemented.
[0011] Figure 2 A simplified exemplary system for providing content suggestions in an application is depicted.
[0012] Figure 3 is an exemplary model architecture for a text-to-content model.
[0013] Figure 4 is used in Figure 3Example model architecture for an unsupervised learning model with pretrained embeddings used in the Model architectures for .
[0014] Figure 5 is a flow chart depicting an exemplary method for providing an unsupervised learning model to predict masked words in an unordered dataset and a pre-trained embedding layer.
[0015] Figure 6 is a flow chart depicting an exemplary method for training a text-based content suggestion model.
[0016] Figure 7 is a block diagram illustrating an exemplary software architecture, portions of which may be used in conjunction with the various hardware architectures described herein.
[0017] Figure 8 is a block diagram illustrating components of an exemplary machine configured to read instructions from a machine-readable medium and perform any of the features described herein. DETAILED DESCRIPTION
[0018] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. After reading this specification, it will be apparent to one of ordinary skill that various aspects may be practiced without using these details. In other instances, well-known methods, processes, components, and / or circuits are described at a relatively high level without being described in detail to avoid unnecessarily obscuring various aspects of the present teachings.
[0019] Many applications commonly used by typical computer users may include content other than text for entering data in the application. New content is often added to these applications, and each content may include a large amount of different content, such as images, icons, emoticons, etc. As a result, it is usually difficult and time-consuming for users to review a large amount of content to find content related to the document the user is working on. Therefore, users may often give up including such content in their documents. However, the additional content may play an important role in document creation. For example, the presentation of content including such as icons and images can convey the message more clearly and is more attractive and more presentable.
[0020] Text-to-content services can provide users with a simple way to select content (e.g., images, icons, emoticons, or keywords) corresponding to the input text by presenting a list of suggestions to the user based on the user's input text. However, it may be difficult to train a model that correctly provides text-to-content suggestions because the training set may be partially filtered to ensure the privacy of the user. In addition, during the training data collection, in each slide in the presentation, only unigrams (e.g., words) in a fixed vocabulary may be recorded. Further, the words in the sentence may be randomly arranged (shuffled) to ensure that the original text cannot be restored. However, most of the currently used text analysis algorithms are based on ordered sentences and data that include correct text sequences. In the absence of correct text sequences, these algorithms may not work.
[0021] To make things more complicated, new content (e.g., icons, images, or emojis) may be frequently introduced in a given application. When new content is provided, the text-to-content model used in the service may need to be retrained to ensure that the content is included in the suggestions. However, because new content is often not fully labeled while old content is, using traditional methods to train a text-to-content model on such an imbalanced dataset may result in a model that rarely predicts new content.
[0022] Text-to-content services typically have several models, each of which suggests certain types of content based on the user's text input, such as text-to-image, text-to-icon, and text-to-emoji. Traditionally, each of these models uses their own methods to train the model. For example, some models use fastText classifiers, while others use tree-based models or a combination of models. These models can use known pre-trained word embeddings to transfer input text to embeddings, and then use their own labeled data to optimize the model through iteration. This approach is not scalable when new content is added to each model frequently. In addition, the individual models may face unbalanced data issues because some models may have a large amount of labeled data while others do not. Generally, the more labeled data available, the better the quality of the suggestions made. As a result, having unbalanced labeled data may lead to unbalanced content suggestions. For example, because the text-to-image model has abundant labeled data, it can have the best quality. On the other hand, the text-to-icon model may not provide high-quality suggestions due to limited labeled data.
[0023] To address these and other problems, in one example, the present specification provides a technique for intelligently suggesting content based on input text by utilizing a deep neural network model. In one example, the deep neural network may include a first layer as an average pooling layer and a second layer as a fully connected layer. The deep neural network model may be trained using supervised training by utilizing a data set including new content and old content. However, in order to ensure improved accuracy of suggestions, the model may be pre-trained by using an unsupervised pre-training mechanism of large-scale unlabeled data. Large-scale unlabeled data may include application domain-specific data and external data received by an application. Domain-specific data may mask certain words and include disordered sentences. Before training a deep neural network model to suggest content, an unsupervised pre-training mechanism may be used to predict masked words and pre-train the first and second layers of the deep neural network model. Unsupervised learning techniques may be used to generate several domain-specific pre-trained word embedding sets (e.g., size 100, 300, 500, 1000, etc.) for a text-to-content service. These word embeddings may incorporate domain-specific knowledge of all user texts in the service. Each text-to-content task can then select the right set of domain-specific pre-trained word embeddings based on its available labeled data, target accuracy, and resource constraints. Thus, this solution provides an improved way to train text-to-content models to improve accuracy and efficiency.
[0024] As will be appreciated by those of ordinary skill in the art upon reading this disclosure, the benefits and advantages provided by these implementations may include, but are not limited to: solutions to the technical problems of inefficient training and providing inaccurate and unbalanced text-to-content suggestions. The technical solutions and implementations provided herein optimize the process of training text-to-content models, improve the accuracy of suggested content, and ensure that new content is included in the suggested content. The benefits provided by these solutions provide a more user-friendly application, improved accuracy, and increased system and user efficiency.
[0025] In general, the methods and systems described herein may include or otherwise utilize machine-trained models to identify content associated with text. Machine learning (ML) generally involves various algorithms that can automatically learn over time. The foundations of these algorithms are generally based on mathematics and statistics, which can be used to predict events, classify entities, diagnose problems, and model function approximations. For example, data generated by an ML model can be used to train a system to identify patterns in user activities, determine the association between individual words and content (e.g., icons, images, or emoticons), and / or identify suggested content associated with a given user input text. This determination can be made after accumulating, reviewing, and / or analyzing user data from a large number of users over time, which can be configured to provide an initial or ongoing training set to an ML algorithm (MLA). In addition, in some implementations, a user device can be configured to send data captured locally during the use of a related application to a cloud or local ML program, and provide supplementary training data that can be used to fine-tune or increase the effectiveness of the MLA. The supplementary data can also be used to facilitate the identification of content and / or increase the training set for future application versions or updates to current applications.
[0026] In different implementations, a training system (which may be referred to as an "ML model trainer") may be used that includes an initial ML model, wherein the initial ML model is configured to generate a subsequently trained ML model from training data obtained from a training data warehouse or from device-generated data. The generation of the ML model may be referred to as "training" or "learning". The training system may include and / or have access to a large amount of computing resources (e.g., a cloud) for training, including many computer server systems suitable for machine learning training. In some implementations, the ML model trainer is configured to automatically generate multiple different ML models for comparison based on the same or similar training data. For example, different basic ML algorithms such as, but not limited to, decision trees, random decision forests, neural networks, deep learning (e.g., convolutional neural networks), support vector machines, regression (e.g., support vector regression, Bayesian linear regression, or Gaussian process regression) may be trained. As another example, the size or complexity of the model may vary between different ML models, such as the maximum depth of a decision tree, or the number and / or size of hidden layers in a convolutional neural network. As another example, different training methods can be used to train different ML models, such as, but not limited to: the selection of training sets, validation sets, and test sets of training data, the ordering and / or weighting of training data items, or the number of training iterations. One or more of the obtained multiple training ML models can be selected based on factors such as, but not limited to, accuracy, computational efficiency, and / or power efficiency. In some implementations, a single trained ML model can be generated.
[0027] The training data may be continuously updated, and one or more of the models used by the system may be modified or regenerated to reflect updates to the training data. Over time, the training system (whether stored remotely, locally, or both) may be configured to receive and accumulate more and more training data items, thereby increasing the amount and variety of training data available for ML model training in order to improve the accuracy, effectiveness, and robustness of the trained ML models.
[0028] Figure 1 An exemplary system 100 is shown on which aspects of the present disclosure may be implemented. System 100 may include a server 110 that may be connected to or include a data store 112 that may be used as a repository in which data sets related to training models may be stored. Server 110 may operate as a shared resource server located in an enterprise that may be accessed by various computer client devices (e.g., client device 130). The server may also operate as a cloud-based server for providing text-to-content services in one or more applications such as application 122.
[0029] Server 110 may include and / or execute text-to-content service 114, which may provide intelligent text-to-content suggestions for a group of users utilizing applications on their client devices (e.g., client device 130). Text-to-content service 114 may be used to check the data input by a user via an application (e.g., application 122 or application 136), and suggest content corresponding to the input data by utilizing various models. In one example, text-to-content service 114 may suggest corresponding content utilizing text-to-image model 116, text-to-icon model 118, and text-to-emoticon model 120. For example, text-to-image model may be used to find images corresponding to user data in an application. Similarly, text-to-icon model may be used to provide suggestion icons corresponding to the input data, and text-to-emoticon model may be used to find emoticons related to the input data. Other models may also be used. For example, text-to-content service may include text-to-keyword model, which suggests keywords corresponding to the text input.
[0030] Each model used as part of the text-to-content service may be trained by a training mechanism 124. The training mechanism 124 may provide a training dataset and perform initial training for each of the models 116, 118, and 120 using a training dataset stored in the data store 112. In one implementation, the training mechanism 124 may use unlabeled training data from the data store 122 (e.g., stored user input data) to train each of the models 116, 118, and 120 via a deep neural network model. The initial training may be performed in an offline phase.
[0031] Client device 130 can be connected to server 110 via network 140. Network 140 can be a wired or wireless network or a combination of wired and wireless networks that connects one or more elements of system 100. Client device 130 can be a personal or handheld computing device having or connected to input / output elements that enables a user to interact with various applications (e.g., application 122 or application 136). Examples of suitable client devices 130 include, but are not limited to, personal computers, desktop computers, laptop computers, mobile phones; smart phones; tablet devices; phablets; smart watches; wearable computers; gaming devices / computers; televisions, etc. About Figure 7 and Figure 8 The internal hardware structure of the client device is discussed in more detail.
[0032] The client device 130 may include one or more applications 136. Each application 136 may be a computer program executing on the client device that configures the device to respond to user input to allow the user to interactively enter data into the application 136. Examples of suitable applications include, but are not limited to, word processing applications, presentation applications, note-taking applications, and communication applications.
[0033] In some examples, the application for receiving user input and responsively providing content suggestions may be executed on the server 110 (e.g., the application 122) and provided via an online service. In one implementation, the web application may communicate with a user agent 132 (e.g., a browser) executing on the client device 130 via a network 140. The user agent 132 may provide a user interface that allows a user to interact with the application 122, and may enable the application 122 to provide user data to the data store 112 for storage as training data. In other examples, the application for receiving user input and providing content suggestions may be a local application, such as an application 136 that is stored and executed on the client device 130 and provides a user interface that allows a user to interact with the application. User data from the application 136 may also be provided to the data store 112 via the network 140 for use by the training mechanism 124.
[0034] Figure 2 A simplified example system 200 for providing content suggestions in an application is shown. In one implementation, the application can receive user input via a user interface 210. The input data can be a portion of text entered on a page of the application (e.g., a slide of a presentation or a page of a word document). The input data can include a single word or any practical number of words, from which feature data can be extracted and input to a text-to-content service 114, which uses training models 116, 118, and 120 to provide suggested images, suggested icons, suggested emoticons, and the like. The text-to-content service can be provided by a remote server. Alternatively, a restricted training model can be available locally to the application to provide some suggested content (e.g., when offline).
[0035] The suggested icons may be processed by the frontdoor and backend unit 230, which may handle the layout and priority of the suggested content when it is presented to the user in a user interface element (e.g., user interface element 240). In one implementation, each of the training models 116, 118, and 120 may assign a score to each suggested content (e.g., each suggested icon) based on the input text, and may present the highest ranked suggested content to the user. Thus, once the user enters a set of data in the application, highly relevant content may be presented to the user for easy selection.
[0036] Figure 3An exemplary simplified architecture 300 for a text-to-content suggestion model is depicted. In one implementation, the text-to-content suggestion model is a deep neural network with two layers. In one example, the input to the text-to-content suggestion model can be a portion of text called a query. Figure 3 As shown in , an exemplary query of the input data set may include word 1 to word n. The query may be a set of words derived from a page of an application providing text to content services (eg, a slide of a presentation or a page of a word document).
[0037] The query can be fed as input into the text-to-content model and used to derive the average pooling layer 310 of the sentence from which the query originates. This can be done by computing the average of the word embeddings in the query. Assume that a given query is represented by Q and is represented as Q = (w1, w2, ..., w n ), and the word embedding matrix is obtained by W E If , then the average value of the word embeddings in Q can be represented by q and calculated as follows:
[0038]
[0039] In the above formula (1), w i A one hot encoded vector may be represented. As is known in the art, a one hot encoded vector may be used to perform binarization of the matrix to properly train the model.
[0040] The word embedding matrix W can be learned on the fly E , or you can use one or more known pre-trained word embeddings to initialize the word embedding matrix W E . In one implementation, the word embedding matrix can be initialized using pre-trained word embeddings such as: Global Vectors for Word Representation (GloVe), Word2vec, or fastText. As is well known in the art, GloVe is an unsupervised learning algorithm for obtaining vector representations of words. Training in Gove can be performed on aggregated global words to perform word co-occurrence statistics from a corpus, and the representation obtained can show a linear substructure of the word vector space. Word2vec refers to a set of related neural network models that are trained to reconstruct the language context of words. Similarly, fastText is a word embedding library created to obtain vector representations of words. In one example, fastText provides a 300-dimensional vector representation of words.
[0041] Once the average of the word embeddings in Q is calculated, an average pooling layer 310 of the sentence from which the input text (e.g., query) originates can be generated. Subsequently, a fully connected layer 320 can be developed for the query. As is known in the art, a fully connected layer in a neural network connects each neuron in one layer of the model to each neuron in another layer. The size of the fully connected layer 320 can depend on the complexity of the task. In one example, the size can vary between 500 and 1000. In one implementation, the fully connected layer 320 can have an activation function such as a rectified linear unit (ReLU) activated on it. This can be done by using the following function:
[0042] h= max(0,W q + b) (2) In equation (2), h may represent the hidden layer output and b may represent the bias in the hidden layer. In addition, a dropout process may be used to reduce overfitting in the model by preventing complex co-adaptation of the training data. ReLU and Dropout are well-known mechanisms, the details of which are readily available in the art.
[0043] Once the fully connected layer 320 is generated and any desired functions are applied to it, a probability distribution can be calculated over all available content (e.g., all icons available for suggestion) to predict the correct icon for the query. This can be calculated using the following formula:
[0044] p(y i | Q) = softmax(h) (3)
[0045] In the above formula (3), p(y i |Q) can represent the probability distribution, y i The probability of the ith icon may be represented by . In this way, the icons most likely to be associated with the query (eg, text in a document) may be predicted as the predicted class 330. These icons may be suggested to the user via a user interface.
[0046] To properly train a text-to-content model, a combination of unsupervised and supervised learning models can be used. Figure 4An exemplary simplified architecture 400 of an unsupervised learning model for pre-training a text-to-content suggestion model is depicted. In one implementation, the unsupervised learning model is a deep neural network, wherein the deep network is trained to have an embedding layer corresponding to at least one of a word or a sentence. It should be understood that the deep neural network can be trained to obtain unstructured text embeddings. These embeddings can provide a basis for supervised learning. The embedding layer can be learned from an unlabeled dataset. In one example, the dataset can be a dataset of unlabeled stored user data obtained by an application over a period of time. Thus, the dataset can be a set of domain-specific unlabeled data that has domain knowledge of the application. As described above, to ensure privacy, the stored user data may not include all words included in the original input text. For example, a filtering method can be used to filter out certain words. In addition, due to constraints on the size of the dictionary used in the training algorithm, not all words in the user input can be recorded. In one example, approximately 30% of the words in a sentence can be randomly masked. In addition, the words in a sentence can be randomly arranged to change their order, making it more difficult to infer the original user input. Thus, the dataset can include a set of unordered words, some of which are masked. As described above, the stored user data may not include all words included in the original input text. For example, a filtering method can be used to filter out certain words. In addition, due to constraints on the size of the dictionary used in the training algorithm, not all words in the user input can be recorded. In one example, approximately 30% of the words in a sentence can be randomly masked. In addition, the words in a sentence can be randomly arranged to change their order, thereby making it more difficult to infer the original user input. Thus, the dataset can include a set of unordered words, some of which are masked. Figure 4 As shown in , an exemplary query of the input dataset may include word 1 to word n, where n is the length of the query, and some words are masked. One advantage of using an unlabeled dataset is that the vocabulary size can be increased by using an external dataset (e.g., the English Gigabyte dataset).
[0047] The unlabeled dataset can then be used to derive the average pooling layer 310 of the model architecture for the query. This can be done by computing the average of the word embeddings in the query. The query can be represented as And when masking w2, it is expressed as In this case, w2 can be replaced by a special token. Then, E Represented by the word embedding matrix, using the query To infer w2. As mentioned above, the word embedding matrix can be initialized with one or more known pre-trained word embeddings (such as GloVe, Word2vec, or fastText).
[0048] Once the word embedding matrix W E Initialized, query The representation (can be represented by to indicate) can be The average of the word embeddings can be calculated as follows:
[0049]
[0050] Then, an average pooling layer 410 may be generated for the sentence from which the query originates. Furthermore, once the average pooling layer 410 is generated, the average pooling layer 410 may be generated based on To develop the fully connected layer 420. As described above, the fully connected layer 420 can have an activation function such as a rectified linear unit (ReLU) activated on it, and a dropout mechanism can be applied to reduce overfitting in the model. Then, a probability distribution can be calculated for all the words in the dataset to predict the unmasked words in the query. This can be calculated using the following formula:
[0051] In formula (5), can represent a probability distribution and y i The probability that the i-th word is a masked word can be represented. In this way, the masked word can be predicted in the prediction class 430. In this way, the unsupervised learning model can function similarly to an autoencoder.
[0052] In one implementation, the unsupervised training model can continue to run to process all the data in the unlabeled dataset. The dataset can include a large number of queries. In one example, the dataset can include tens of thousands of queries, and the unsupervised training model can take several days to train. Once the dataset has been processed by the algorithm, an embedding layer can be generated, which can be further used as a pre-trained embedding layer to train the text-to-content model. The pre-trained embedding layer can be used as an embedding dictionary that is specifically tailored for the data in the application used to derive the input dataset. Once the unsupervised learning model 400 is completed, the final pre-trained embedding generated as part of the process can be used to train one or more text-to-content models. For example, the pre-trained embedding layer can be used to train all text-to-content models provided in the text-to-content service.
[0053] Figure 5 is a flow chart depicting an exemplary method 500 for providing an unsupervised learning model to predict masked words in an unordered data set. The method 500 can start at 505 and proceed to accessing a data set at 510. As described above, the data set can be an application-specific data set based on user data that is received from the application and recorded in a data set designed for use by a training mechanism. The database can be a large-scale data set of unordered queries containing masked words. The size of the data set can be determined by the size constraints of the dictionary designed for the training algorithm and can vary depending on the application. In order to train the model, the method 500 can process all queries in the data set.
[0054] After accessing the dataset to retrieve the query, method 500 can proceed to initializing the training model using one or more general pre-trained embeddings at 515. General pre-trained embeddings can be industry standard pre-trained embeddings that use dictionary words as their input. These pre-trained embeddings are different from the pre-trained embeddings generated by the unsupervised training model disclosed herein in that the pre-trained embeddings generated by the disclosed unsupervised training model are domain-specific and are trained using application-specific data. Method 500 can utilize one or more pre-trained embeddings to initialize the word embedding matrix of the query.
[0055] Once the training model is initialized using one or more pre-trained embeddings, method 500 may proceed to generating an average pooling layer for the sentence from which the query originated at 520 before generating a fully connected layer at 525. Then, at 530, an activation function such as ReLU may be applied to the fully connected layer. Then, method 500 may proceed to applying a dropout mechanism to the fully connected layer at 535, and then at 540, the neural network may predict masked words. The steps of method 500 may be repeated for each query in the dataset to generate predicted classes.
[0056] During this unsupervised learning process, pre-trained embeddings are generated. Pre-trained embeddings are application-specific embeddings that can be used throughout the text-to-content service to improve the quality of the text-to-content model and share domain-specific knowledge across text-to-content models. In one implementation, several groups of pre-trained embedding layers can be generated by an unsupervised learning algorithm, where each group of pre-trained embedding layers has a different dimension. For example, a first embedding layer with a 100-dimensional vector can be generated. This may be useful for training algorithms with smaller training data and therefore relatively simple training algorithms. A second pre-trained embedding layer with a 500-dimensional vector can be generated for training more complex models. A third group of pre-trained embedding layers with a 1000-dimensional vector can be created for training highly complex models. Each of these embedding layers can have corresponding initialization weights that can be used to train downstream text-to-content models, as described below.
[0057] Figure 6 6 is a flow chart depicting an exemplary method 600 for training a text-to-content based suggestion model. Method 600 may begin at 605 and proceed to accessing a training dataset at 610. The dataset used at this stage may be a labeled dataset provided specifically for training a specific type of text-to-content suggestion model (e.g., text-to-icon, text-to-image, or text-to-emoji). Thus, the training performed at this stage may be supervised learning. Depending on the type of training required (e.g., initial training or retraining due to new content), the size of the dataset used may be relatively small and may depend on the type of model being trained.
[0058] After accessing the dataset to access the query, method 600 can proceed to initializing the training model using a pre-trained embedding layer generated by the unsupervised learning model at 615. The pre-trained embedding layer that is most suitable for the type of model being trained can be selected at this stage. Method 600 can use the pre-trained embedding to initialize the word embedding matrix for the query.
[0059] Once the training model is initialized using the pre-trained embeddings, method 600 may proceed to generating an average pooling layer for the sentence from which the query originated at 620 before applying the initialization weights at 625. The initialization weights may be derived from an unsupervised learning model to initialize the text-to-content model being trained. Method 600 may then proceed to generating a fully connected layer at 630. At 635, an activation function such as a ReLU may be applied to the fully connected layer. Method 600 may then proceed to applying a dropout mechanism to the fully connected layer at 640, and then at 645, an output prediction class may be created.
[0060] The steps of method 600 may be repeated for each query in the dataset to generate predicted classes. In one implementation, random weights may be used between the fully connected layer and the output predicted classes. The size of the predicted classes may depend on the size of the content available in the application. For example, if the application includes 800 icons that may be used for a text-to-icon suggestion model, the size of the classes created for the model may be 800. The steps of the method may be repeated to fine-tune the model, each time using different weights to achieve the final model. In this way, the pre-trained embedding layer generated by the unsupervised learning model may be used to train downstream tasks.
[0061] In one implementation, an engineering layer can be added to the final model to ensure that the model functions correctly in an online environment. This is because during testing, the model may function differently than in the real world and online environment, because the signal may hide many other models, etc. The engineering layer can guide the recommendations. In particular, for the new content ratio, the engineering layer can adjust the proportion of new content presented as recommendations.
[0062] When adding new content to an application, other challenges may be presented in training text into a content model. For example, the new content may not include tags, and may use much less data than the old content to promote the new content. Typically, when processing new content, it may be difficult to control the proportion of new content suggested by the model. To address these issues, the training model can use a third-party enhanced data set to guide the new content. For example, Wikipedia and dictionary data sets can be used for this purpose. Other mechanisms can also be used. For example, training can include a decay (dropoff) mechanism that gradually removes old content in each batch of training data. This may cause the ratio of new content to old content in each batch to gradually increase. Another mechanism may be to punish errors in new content more. However, another method that can be used to address these issues is to create a new model for the new content and fuse the results of the new model with the old model. This may involve the problem of making the fusion logic controllable. As previously discussed, an engineering layer can also be added to control the new content output score during online operation.
[0063] It should be noted that the models that provide text-to-content suggestions can be hosted locally on the client (e.g., a text-to-content engine) or remotely in the cloud (e.g., a text-to-content service). In one implementation, some models are hosted locally, while other models are stored in the cloud. This enables the client device to provide some suggestions even when the client is not connected to a network. However, once the client is connected to a network, the application is able to provide better and more complete text-to-content suggestions.
[0064] Therefore, a method and system for generating application-specific embeddings for a text-to-content service is provided. Application-specific embeddings can be used to train text-to-content models to improve the quality of models that may lack sufficient labeled data, and to share domain-specific knowledge across text-to-content models. The improved training method can utilize unsupervised learning techniques and open source pre-trained word embeddings to generate various domain-specific pre-trained word embedding sets for all text-to-content models in the text-to-content service. Domain-specific pre-trained word embeddings can be shared between downstream text-to-content tasks through joint pre-trained word embeddings to guide new text-to-content suggestions and minimize the difference in suggestion quality between different models. As a result of first adopting unsupervised learning models, much less labeled data may be required to train each text-to-content model. In addition, these models generate better suggestions using the same amount of labeled data. This means that these algorithms can use unlabeled data efficiently and can significantly reduce the time required to guide a fairly good quality text-to-content suggestion model. This can improve the overall experience of users, increase efficiency, and help locate relevant content more easily when needed.
[0065] Figure 77 is a block diagram 700 illustrating an exemplary software architecture 702, aspects of which may be used in conjunction with the various hardware architectures described herein that can implement any of the features described above. Figure 7 704 is a non-limiting example of a software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. The software architecture 702 can be executed on hardware such as client devices, native application providers, web servers, server clusters, external services, and other servers. The representative hardware layer 704 includes a processing unit 706 and associated executable instructions 708. The executable instructions 708 represent executable instructions of the software architecture 702, which include implementations of the methods, modules, etc. described herein.
[0066] The hardware layer 704 also includes a memory / storage device 710, which also includes executable instructions 708 and accompanying data. The hardware layer 704 may also include other hardware modules 712. The instructions 708 held by the processing unit 706 may be a portion of the instructions 708 held by the memory / storage device 710.
[0067] Exemplary software architecture 702 can be conceptualized as layers, each of which provides various functions. For example, software architecture 702 may include layers and components such as operating system (OS) 714, library 716, framework 718, application 720, and presentation layer 744. In operation, application 720 and / or other components in a layer may call API calls 724 for other layers and receive corresponding results 726. The layers shown are representative in nature, and other software architectures may include additional or different layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 718.
[0068] OS 714 can manage hardware resources and provide public services. OS 714 can include, for example, kernel 728, service 730 and driver 732. Kernel 728 can serve as an abstraction layer between hardware layer 704 and other software layers. For example, kernel 728 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Service 730 can provide other public services for other software layers. Driver 732 can be responsible for controlling bottom hardware layer 704 or interacting with it. For example, driver 732 can be configured according to hardware and / or software, including display driver, camera driver, memory / storage device driver, peripheral device driver (e.g., through universal serial bus (USB)), network and / or wireless communication driver, audio driver, etc.
[0069] The library 716 can provide a public infrastructure that can be used by the application 720 and / or other components and / or layers. The library 716 generally provides functions for performing tasks for other software modules, rather than directly interfacing with the OS 714. The library 716 may include a system library 734 (e.g., a C standard library), which may provide functions such as memory allocation, string operations, file operations, etc. In addition, the library 716 may include an API library 736, such as a media library (e.g., a library for supporting the presentation and operation of image, sound and / or audio data formats), a graphics library (e.g., an OpenGL library for presenting two-dimensional and three-dimensional graphics on a display), a database library (e.g., SQLite or other relational database functions), and a web library (e.g., WebKit, which may provide web browsing functions). The library 716 may also include a variety of other libraries 738 to provide many functions to the application 720 and other software modules.
[0070] Framework 718 (which is sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by application 720 and / or other software modules. For example, framework 718 can provide various GUI functions, advanced resource management, or advanced location services. Framework 718 can provide a wide range of other APIs for application 720 and / or other software modules.
[0071] Applications 720 include built-in applications 740 and / or third-party applications 742. Examples of built-in applications 740 may include, but are not limited to, contact applications, browser applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 742 may include any application developed by an entity other than the vendor of a particular system. Applications 720 may use the functionality available via OS 714, library 716, framework 718, and presentation layer 744 to create a user interface to interact with a user.
[0072] Some software architectures use virtual machines, such as virtual machine 748. Virtual machine 748 provides an execution environment in which applications / modules can be executed as if they were on a hardware machine (e.g., Figure 8 The virtual machine 748 may be hosted by a host OS (e.g., OS 714) or a hypervisor and may have a virtual machine monitor 746 that manages the operation of the virtual machine 748 and the interaction with the host operating system. The software architecture may be different from the software architecture 702 outside the virtual machine, executing within the virtual machine 748 such as OS 750, libraries 752, frameworks 754, applications 756, and / or presentation layer 758.
[0073] Figure 8800 is a block diagram illustrating components of an exemplary machine 800 that is configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any of the features described herein. The exemplary machine 800 has the form of a computer system in which instructions 816 (e.g., in the form of software components) for causing the machine 800 to perform any of the features described herein can be executed. Thus, the instructions 816 can be used to implement the methods or components described herein. The instructions 816 cause the unprogrammed and / or unconfigured machine 800 to operate as a specific machine configured to perform the described features. The machine 800 can be configured to operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 can operate in the capacity of a server machine or a client machine in a server-client network environment, or operate as a node in a peer-to-peer or distributed network environment. The machine 800 may be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smart phone, a mobile device, a wearable device (e.g., a smart watch), and an Internet of Things (IoT) device. Furthermore, while only a single machine 800 is shown, the term "machine" includes a collection of machines that individually or jointly execute instructions 816.
[0074] The machine 800 may include a processor 810, a memory 830, and I / O components 850, which may be communicatively coupled, for example, via a bus 802. The bus 802 may include multiple buses that couple various elements of the machine 800 via various bus technologies and protocols. In one example, the processor 810 (e.g., including a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 812a to 812n capable of executing instructions 816 and processing data. In some examples, one or more processors 810 may execute instructions provided or identified by one or more other processors 810. The term "processor" includes multi-core processors, which include cores capable of executing instructions simultaneously. Although Figure 8 Multiple processors are shown, but machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, machine 800 may include multiple processors distributed among multiple machines.
[0075] The memory / storage device 830 may include a main memory 832, a static memory 834, or other memory and storage unit 836, both of which can be accessed by the processor 810, for example, via the bus 802. The storage unit 836 and the memories 832, 834 store instructions 816, which include any one or more of the functions described herein. The memory / storage device 830 may also store temporary, intermediate and / or long-term data for the processor 810. The instructions 816 may also reside, during execution, in whole or in part in the memory 832, 834, the storage unit 836, at least one of the processors 810 (e.g., a command buffer or cache memory), or at least one of the I / O components 850, or any suitable combination thereof. Therefore, the memories 832, 834, the storage unit 836, the memory in the processor 810, and the memory in the I / O components 850 are examples of machine-readable media.
[0076] As used herein, a "machine-readable medium" refers to a device that is capable of temporarily or permanently storing instructions and data that cause the machine 800 to operate in a specific manner. As used herein, the term "machine-readable medium" does not cover temporary electrical or electromagnetic signals per se (e.g., on a carrier waveform that propagates through the medium); therefore, the term "machine-readable medium" may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible machine-readable media may include, but are not limited to, non-volatile memory (e.g., flash memory or read-only memory (ROM)), volatile memory (e.g., static random-access memory (RAM) or dynamic RAM), buffer memory, cache memory, optical storage media, magnetic storage media and devices, network accessible devices or cloud storage, other types of storage, and / or any suitable combination thereof. The term "machine-readable medium" applies to a single medium or a combination of multiple media that is used to store instructions (e.g., instructions 816) executed by the machine 800, such that when these instructions are executed by one or more processors 810 of the machine 800, the machine 800 is caused to perform one or more of the features described herein. Thus, a “machine-readable medium” may refer to a single storage device, as well as a “cloud-based” storage system or storage network that includes multiple storage devices or devices.
[0077] I / O components 850 may include a wide variety of hardware components suitable for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 850 included in a particular machine will depend on the type and / or function of the machine. For example, a mobile device such as a mobile phone may include a touch input device, while a headless server or IoT device may not include such a touch input device. Figure 8The specific examples of I / O components shown in are by no means limiting, and other types of components may be included in the machine 800. The grouping of I / O components 850 is merely to simplify this discussion, and such grouping is by no means limiting. In various examples, the I / O components 850 may include a user output component 852 and a user input component 854. The user output component 852 may include, for example, a display component (e.g., a liquid crystal display (LCD) or a projector) for displaying information, an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor or a force feedback device), and / or other signal generators. The user input component 854 may include, for example, an alphanumeric input component (e.g., a keyboard or a touch screen), a pointing component (e.g., a mouse device, a touch pad or other indicating instrument), and / or a tactile input component (e.g., a physical button, or a touch screen that provides the position and / or force of a touch or touch gesture) configured to receive various user inputs (e.g., user commands and / or selections).
[0078] In some examples, I / O components 850 may include biometric components 856, motion components 858, environmental components 860, and / or position components 862, as well as a variety of other environmental sensor components. For example, biometric components 856 may include components for detecting body expressions (e.g., facial expressions, vocal expressions, hand or body gestures, or eye tracking), measuring biosignals (e.g., heart rate or brain waves), and identifying people (e.g., through voice-based, retinal and / or facial recognition). Position component 862 may include, for example, a position sensor (e.g., a global positioning system (GPS) receiver), an altitude sensor (e.g., an air pressure sensor from which altitude can be derived), and / or a direction sensor (e.g., a magnetometer).
[0079] The I / O components 850 may include a communication component 864 that implements a variety of techniques that may be used to couple the machine 800 to a network 870 and / or a device 880 via respective communication couplings 872 and 882. The communication component 864 may include one or more network interface components or other suitable devices for interfacing with the network 870. For example, the communication component 864 may include components suitable for providing wired communication, wireless communication, cellular communication, near field communication (NFC), Bluetooth communication, Wi-Fi, and for providing communication through other modes. The device 880 may include other machines or various peripheral devices (e.g., peripheral devices coupled via USB).
[0080] In some examples, the communication component 864 can detect the identifier or include a component suitable for detecting the identifier. For example, the communication component 864 can include a radio frequency identification (RFID) tag reader, an NFC detector, an optical sensor (e.g., a one-dimensional or multi-dimensional bar code or other optical code), and / or an acoustic detector (e.g., a microphone for identifying an audio signal of a tag). In some examples, the location information can be determined based on information from the communication component 862 (e.g., a geographic location via an Internet Protocol (IP) address, a location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification and / or signal triangulation).
[0081] Although various embodiments have been described, this description is intended to be exemplary rather than restrictive, and it should be understood that more embodiments and implementations may exist within the scope of the embodiments. Although multiple possible feature combinations are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature of any embodiment may be used in combination with any other feature or element in any other embodiment or be substituted with it. Therefore, it should be understood that any feature shown and / or discussed in this disclosure may be implemented together in any appropriate combination. Therefore, the embodiments herein are not limited except for the appended claims and their equivalents. Moreover, various modifications and changes may be made within the scope of the appended claims.
[0082] Generally, the functions described herein (e.g., Figure 1-6 The features shown in (shown in) can be implemented using software, firmware, hardware (e.g., fixed logic, finite state machines and / or other circuits), or a combination of these implementations. In the case of software implementation, when the program code is executed on a processor (e.g., a CPU or a CPU cluster), the specified task is performed. The program code can be stored in one or more machine-readable memory devices. The features of the technology described herein are system-independent, which means that these technologies can be implemented on various computing systems with various processors. For example, an implementation may include an entity (e.g., software) that enables hardware to perform operations (e.g., processor function blocks, etc.). For example, a hardware device may include a machine-readable medium, which may be configured to maintain instructions that enable a hardware device (including an operating system and associated hardware executed thereon) to perform operations. Therefore, instructions can be used to configure an operating system and associated hardware to perform operations, thereby configuring or otherwise adjusting the hardware device to perform the above-mentioned functions. These instructions can be provided through a machine-readable medium through various different configurations of hardware elements that execute instructions.
[0083] Although what is considered to be the best mode and / or other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and these teachings may be applied to many applications, only some of which are described herein. The appended claims are intended to claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.
[0084] Unless otherwise indicated, all measurements, values, levels, positions, amplitudes, sizes and other specifications set forth in this specification (including in the claims that follow) are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they are related and the customary practices in the art to which they belong.
[0085] The scope of protection is limited only by the claims that follow. That scope is intended and should be interpreted to be as broad as consistent with the ordinary meaning of the language used in the claims when interpreted in light of this specification and subsequent prosecution history and to encompass all structural and functional equivalents.
[0086] Except as described above, nothing stated or described is intended or should be construed as causing any component, step, feature, object, benefit, advantage or equivalent to be dedicated to the public regardless of whether it is stated in the claims or not.
[0087] It should be understood that the terms and expressions used herein have the ordinary meaning as is consistent with such terms and expressions as are relevant to the respective areas of inquiry and study to which they correspond unless specific meanings are otherwise indicated herein.
[0088] Only use relational terms such as first and second, etc. to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "include", "comprises" and any other variations thereof are intended to cover non-exclusive inclusions, so that the process, method, article or device including a list of elements not only includes these elements, but also may include other elements not explicitly listed or other elements inherent to such process, method, article or device. Without further limitation, an element preceded by "a" or "an" does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0089] An abstract of the disclosure is provided to enable the reader to quickly determine the nature of the technical disclosure. It should be understood that it is not provided to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description, it can be seen that various features are combined together in various examples in order to simplify the disclosure. This approach to the disclosure should not be interpreted as reflecting the intention that any claim requires more features than those clearly stated in the claim. On the contrary, as reflected in the attached claims, the subject matter of the invention is less than all the features of a single disclosed embodiment. Therefore, the attached claims are incorporated into the detailed description, wherein each claim item exists independently as a separate subject matter.
Claims
1. A data processing system, comprising: processor; as well as A memory in communication with the processor, the memory comprising executable instructions, which, when executed by the processor, cause the data processing system to perform the following operations: accessing a dataset comprising unlabeled training data collected from user usage of an application that provides text-to-content suggestions by suggesting content in response to a text query; applying an unsupervised training process to the dataset to generate a plurality of pre-trained embeddings; as well as The plurality of pre-trained embeddings generated by the unsupervised training process are utilized in a supervised training process, the supervised training process using a labeled dataset to train a text-to-content suggestion machine learning (ML) model utilized by the application, the labeled dataset being smaller than the dataset comprising unlabeled data.
2. The data processing system according to claim 1, wherein: Each of the unsupervised training process and the supervised training process includes an average pooling layer and a fully connected layer.
3. The data processing system according to claim 1, wherein: The instructions also cause the processor to apply initialization weights to the supervised training process, the initialization weights being obtained through the unsupervised training process.
4. The data processing system according to claim 1, wherein: The instructions also cause the processor to apply an engineering layer to the trained text-to-content suggestion ML model to control a proportion of new content suggested.
5. The data processing system according to claim 1, wherein: The unlabeled training data includes an unordered list of words.
6. The data processing system according to claim 5, wherein: The unordered word list includes at least one masked word; and The unsupervised training process generates predicted words corresponding to the masked words.
7. The data processing system according to claim 1, wherein: The unsupervised training process generates multiple pre-trained embedding layers, each pre-trained embedding layer has a different vector dimension for use in conjunction with a different text-to-content suggestion model.
8. A method for training a text-to-content suggestion machine learning (ML) model, the method comprising: accessing a dataset comprising unlabeled training data collected from user usage of an application that provides text-to-content suggestions by suggesting content in response to a text query; applying an unsupervised training process to the dataset to generate a plurality of pre-trained embeddings; as well as The plurality of pre-trained embeddings generated by the unsupervised training process are utilized in a supervised training process, the supervised training process using a labeled dataset to train a text-to-content suggestion ML model utilized by the application, the labeled dataset being smaller than the dataset comprising unlabeled data.
9. The method according to claim 8, further comprising: Initialization weights are applied to the supervised training process, the initialization weights being obtained through the unsupervised training process.
10. The method according to claim 8, further comprising: Apply an engineering layer to the trained text-to-content recommendations ML model to control the proportion of new content suggested.
11. The method according to claim 8, wherein: The unsupervised training process generates multiple pre-trained embedding layers, each pre-trained embedding layer has a different vector dimension for use in conjunction with a different text-to-content suggestion model.
12. A non-transitory computer readable medium having stored thereon instructions which, when executed, cause a programmable device to: accessing a dataset comprising unlabeled training data collected from user usage of an application that provides text-to-content suggestions by suggesting content in response to a text query; applying an unsupervised training process to the dataset to generate a plurality of pre-trained embeddings; as well as The plurality of pre-trained embeddings generated by the unsupervised training process are utilized in a supervised training process, the supervised training process using a labeled dataset to train a text-to-content suggestion machine learning (ML) model utilized by the application, the labeled dataset being smaller than the dataset comprising unlabeled data.
13. The non-transitory computer readable medium of claim 12, wherein: The instructions also cause the programmable device to apply initialization weights to the supervised training process, the initialization weights being obtained through the unsupervised training process.
14. The non-transitory computer readable medium of claim 12, wherein: The unlabeled training data includes an unordered list of words.
15. The non-transitory computer readable medium of claim 14, wherein: The unordered word list includes at least one masked word; and The unsupervised training process generates predicted words corresponding to the masked words.
