Evaluate similarity between items using embeddings produced using a distributed training framework

Abstract

A resource-efficient technique for generating and utilizing a set of trained embeddings is described. Regarding its training phase, the technique receives a set of high-dimensional training samples with sparse representations. The technique uses a distributed training framework on a computing device to process the training samples. Regarding its inference phase, the technique utilizes the embeddings generated by the training framework. However, in one implementation, the inference phase processes a prediction function that differs from the prediction function used by the training framework. One implementation of the inference phase processing involves determining the distance between the query embedding and candidate item embeddings, where each such embedding is obtained or derived from the trained embeddings generated by the training framework. Another manifestation of the inference phase processing involves adjusting click counts based on relationships between identified item embeddings.

Description

Background Technology

[0001] Modern search engines typically use machine-trained models to help determine whether candidate items are suitable matches for a submitted query. Candidate items can correspond to documents, images, videos, etc. However, some of these machine-trained models are also highly complex, as evidenced by the fact that they include a large number of neural network layers and parameters. The training framework may require significant computational resources to generate such complex models, especially when the training samples include high-dimensional vectors. The inference-stage processing system may also require significant computational resources to run complex models. The complexity of the model can also adversely affect the latency of the inference-stage processing system in processing user-submitted queries. Summary of the Invention

[0002] This paper describes a resource-saving technique for generating and utilizing a set of trained embeddings. During its training phase, the technique uses a distributed training framework to generate trained embeddings by operating on a set of training samples. More specifically, the distributed training framework divides the training task into multiple independent (or largely independent) tasks, which are then executed in parallel. The use of a distributed training framework is particularly helpful in allowing the technique to efficiently operate on training samples represented by high-dimensional and sparse expressions. In one implementation, each training sample identifies a specific query, a candidate option, and an indication of whether that option has been selected by at least one user in response to a submitted query.

[0003] Regarding its inference phase, this technique relies on embeddings already generated by the training framework. However, in one implementation, the inference phase processing applies a different prediction function compared to the one used by the training framework. This inference phase processing avoids the use of complex machine-trained models and high-dimensional embeddings, which is one factor allowing the inference phase processing to be performed in a faster and more resource-efficient manner.

[0004] In one application, the push-out phase involves determining a similarity score, which describes how well the considered candidates are a suitable match for the user-submitted query. The determination process sequentially includes: identifying query embeddings associated with one or more terms in the query; identifying item embeddings associated with candidates; and determining the distance between the query embeddings and item embeddings in the vector space. The query embeddings and item embeddings are obtained or derived from trained embeddings already generated by the training framework. For example, this technique can generate query embeddings by identifying query component embeddings associated with terms in the query, and then summing these query component embeddings.

[0005] In another application, the push-out phase involves adjusting the number of clicks a given item is considered to have received based on a weighted combination of clicks received by other items, where all such clicks are made in response to a given query submission. This technique computes a weighted value for each pair of items by identifying the embeddings associated with the items and then determining the similarity between the embeddings. This push-out phase has the net effect of transferring clicks from popular documents to less popular documents, provided that the embeddings associated with those documents express similar intent. This technique can use embeddings generated by the training framework described above or any other embedding generation mechanism.

[0006] The technologies summarized above can be manifested in various types of systems, devices, components, methods, computer-readable storage media, data structures, graphical user interface presentations, artifacts, etc.

[0007] This summary is provided to introduce some concepts in a simplified form; these concepts will also be described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0008] Figure 1 An overview of the training environment for generating trained embeddings and the inference stage environment for applying trained embeddings is shown.

[0009] Figure 2 It shows that it can be made by Figure 1 The training framework processes illustrative training samples.

[0010] Figure 3 It was shown as Figure 1 The diagram shows an overview of a portion of the distributed training framework for the training environment.

[0011] Figure 4 It shows Figure 3 An example of how the training framework operates.

[0012] Figure 5 It shows in Figure 1 and Figure 3 An example of the processing logic used in the computing device within the training framework.

[0013] Figure 6 An implementation of a similarity evaluation engine for determining the relevance of a query to its candidates is shown. The similarity evaluation engine is... Figure 1 It is an element of the reasoning stage environment.

[0014] Figure 7 This is demonstrated by showing embeddings in a vector space. Figure 6 The operation of the similarity evaluation engine.

[0015] Figure 8 This shows how to adjust the click count associated with an item. Figure 1 The implementation of a similarity assessment engine.

[0016] Figure 9 This is demonstrated by showing embeddings in a vector space. Figure 8 The operation of the similarity evaluation engine.

[0017] Figure 10 It shows the provision Figure 1 This is an overview of the process of operating a training phase environment.

[0018] Figure 11 It shows the provision Figure 1 , Figure 6 and Figure 8 This is an overview of the operation process of a similarity assessment engine.

[0019] Figure 12 It shows the provision Figure 8 This outlines the process of a specific similarity assessment engine. The similarity assessment engine adjusts the click count information in the data structure.

[0020] Figure 13 An illustrative type of computing device is shown that can be used to implement any aspect of the features shown in the foregoing figures.

[0021] The same reference numerals are used throughout the specification and drawings to denote the same components and features. The numerical designations of Series 100 initially appeared in... Figure 1 The features found in the series 200, the numerical designation originally in Figure 2 The features found in the series 300, the numerical designation originally in Figure 3 The features discovered in it, and so on. Detailed Implementation

[0022] This disclosure is organized as follows. Part A describes a computer-implemented environment for generating and applying trained embeddings. Part B elaborates on illustrative methods for interpreting the operation of the computational environment of Part A. And Part C describes illustrative computational functions that can be used to implement any aspect of the features described in Parts A and B.

[0023] As a reference, the term "hardware logic circuit" corresponds to the technology of including one or more hardware processors (e.g., CPU, GPU, etc.) that execute machine-readable instructions stored in memory and / or one or more other hardware logic units (e.g., FPGA) that perform operations using a task-specific set of fixed and / or programmable logic gates. Part C provides additional information about an implementation of the hardware logic circuit. In some contexts, each of the terms "component," "module," "logic," "engine," "system," and "tool" refers to a portion of the hardware logic circuit that performs a specific function.

[0024] In one case, the separation of the various components into different units shown in the figures may reflect the use of corresponding different physical and tangible parts in the actual implementation. Alternatively or additionally, any single component shown in the figures may be implemented by multiple actual physical parts. Alternatively or additionally, the depiction of any two or more separate components in the figures may reflect different functions performed by a single actual physical part.

[0025] The other accompanying figures illustrate the concepts in flowchart form. In this form, certain operations are described as constituting different blocks executed in a certain order. Such implementations are illustrative and not restrictive. Some blocks described herein may be grouped together and executed in a single operation, some blocks may be divided into multiple component blocks, and some blocks may be executed in a different order than that shown herein (including in the form of parallel execution of these blocks). In one implementation, the blocks shown in the flowcharts concerning processing-related functions can be considered as corresponding logical components and can be implemented by the hardware logic circuitry described in Part C, which in turn can be implemented by one or more hardware processors and / or other logic units comprising a task-specific set of logic gates.

[0026] Regarding the terminology, the phrase "configured to" encompasses various physical and tangible mechanisms for performing the identified operations. These mechanisms can be configured to perform the operations using hardware logic circuitry from part of C.

[0027] Any storage resource or any combination of storage resources described herein may be considered a computer-readable medium. In many cases, a computer-readable medium refers to some form of physical and tangible entity. The term computer-readable medium also includes, for example, propagated signals transmitted or received via physical conduits and / or air or other wireless media. However, the specific term "computer-readable storage medium" explicitly excludes the propagated signal itself, while including all other forms of computer-readable medium.

[0028] The following description may identify one or more features as “optional.” Statements of this type should not be construed as an exhaustive indication of features that could be considered optional; that is, other features may be considered optional, even if not explicitly indicated in the text. Furthermore, any description of a single entity is not intended to exclude the use of multiple such entities; similarly, a description of multiple entities is not intended to exclude the use of a single entity. Moreover, while this specification may interpret certain features as alternative ways of performing the identified function or implementing the identified mechanism, these features may also be combined in any combination. Furthermore, the term “multiple” refers to two or more items and does not necessarily imply “all” items of a particular type, unless explicitly stated otherwise. Furthermore, unless otherwise stated, descriptors such as “first,” “second,” “third,” etc., are used to distinguish different items and do not imply an ordering among the items. Finally, the terms “exemplary” or “illustrative” refer to one of a potential plurality of implementations.

[0029] A. Descriptive computing environment

[0030] Figure 1 An overview of the training phase environment 102 for generating trained embeddings and the inference phase environment 104 for applying the trained embeddings generated by the training phase environment 102 is shown. The following description will begin with an overview of the training phase environment 102 and the inference phase environment 104. Additional details about the individual components in these two environments (102, 104) will then be provided.

[0031] Starting from training phase environment 102, sample mining system 106 generates a set of training samples processed by training framework 108. In one implementation, sample mining system 106 generates at least a portion of each training sample based on one or more logs generated by search engine 110 and stored in data repository 112. A training log corresponds to a "click log," which provides information about queries submitted by end users during multiple search sessions and the items selected by users in response to these queries. For example, an entry in the click log may indicate that at least one user submitted the query "funny dog ​​video." The click log also indicates the specific video selected by the user in response to submitting the query "funny dog ​​video." For example, in response to this query, search engine 110 may provide the user with a search results page. The user may have clicked a link on the search results page associated with a video having a name similar to the query, such as "video of a funny dog." In other cases, the selected item may correspond to other types of content, such as, but not limited to, documents, images, advertisements, audio files (e.g., songs), etc. Click logs can use any type of item identifier to represent the selected item, such as a Uniform Resource Locator (URL) assigned to the selected item.

[0032] Additionally or alternatively, the sample mining system 106 may provide other types of examples of relationships between queries and documents. For example, sample mining 106 may provide a platform that allows a group of human analysts to manually indicate whether a given document is relevant to a given query for each of multiple documents. In other cases, the sample mining system 106 may more generally identify relationships between actions performed by at least one user at one or more previous moments. For example, the sample mining system 106 may extract samples from action logs indicating an item selected by a user after selecting another item. However, for simplicity, the following description will be based on the assumption that the sample mining system 106 relies on the type of click log described above, which provides a record of items selected by a user in response to submitting a specific query.

[0033] The sample mining system 106 can filter click log information to exclude potentially low-quality training samples. For example, the sample mining system 106 can exclude items that have been clicked a large number of times for different queries, where "large" is evaluated using a threshold specific to the environment. Furthermore, the sample mining system 106 can exclude samples that do not have a sufficient number of clicks for a given query, again where "low" is evaluated using a threshold specific to any given environment.

[0034] The sample mining system 106 can formulate each training sample as a series of vectors. Let's move forward for now. Figure 2The figure illustrates an illustrative training sample generated based on information from click logs. It instructs at least one user to submit the query "funny dog ​​video" and, in response, click the item with identifier D3. The sample mining system 106 can represent each word in the query using one-hot vectors. Alternatively, the sample mining system 106 can represent each n-gram (e.g., each ternary gram) in the query as a one-hot vector. For example, the sample mining system 106 can represent the query "funny dog ​​video" using one-hot vectors associated with the ternary gramms "\tfu", "fun", "unn" and "nny", "ny_", "y_d", "_do", "dog", etc., respectively, where "t\" is the symbol indicating the start of the query and "_" indicates a space between grammatical units. The sample mining system 106 can represent each such ternary gram as a one-hot vector. The sample mining system 106 can represent item D3 as another one-hot vector. A one-hot vector corresponds to a vector that dedicates a dimension to each word in the vocabulary, and this vector stores "1" entries provided in the dimension associated with the specific item being represented, and "0" entries for all other items in the vocabulary. For example, sample mining system 106 can represent the word "funny" by storing "1" entries in the dimension associated with the word "funny" and "0" entries for the other items.

[0035] It is understandable that, in general, training sample 202 has a relatively large size. For example, each item vector individually has a dimension at least as large as the total number of items that can be selected in the main set of items, which can include millions or more items. Training sample 202 can also be described as sparse, as it includes "1" entries only for those dimensions associated with the terms in the query and the selected item D3; the remaining dimensions are all "0" entries.

[0036] The samples described above are positive samples because the click logs contain evidence that at least one user actually clicked on item D3 after submitting the query "funnydog video". Although not shown, the sample mining system 106 can also generate a set of negative training samples. Each negative training sample includes a query extracted from the search logs and a randomly selected item identifier. In most cases, because the item is randomly selected, it will not be a suitable match for the query. This assumption can be tested by determining whether a positive relationship exists between the randomly selected item and the query; if a positive relationship exists, the pair can be discarded as a viable negative sample.

[0037] Back Figure 1The sample mining system 106 can store at least some training samples in the data repository 114. Then, the training framework 108 uses machine learning to iteratively process the training samples to produce a trained set of embeddings. For example, for... Figure 2 Using the training samples, training framework 108 can generate embeddings for each word and / or each ternary grammar in the query "funny dog ​​video". Training framework 108 can also generate another trained embedding for the identifier D3.

[0038] More generally, the trained embeddings correspond to the factorization or distributed representation of information fragments using vectors of dimension k. The dimension k is an adjustable parameter. In a non-restrictive implementation, k is set to 16. Factorization or distributed embeddings typically represent terms using multiple "1" entries in the corresponding dimension. This contrasts with one-hot vectors, which assign dedicated slots in their vectors to corresponding terms. The training framework 108 stores its set of trained embeddings in a data repository 116.

[0039] As will be explained in more detail below, the training framework 108 uses a distributed collection of computing devices 118. Computing devices 118 may correspond to corresponding servers provided in a single location or distributed across multiple locations. Logically, multiple computing devices 118 implement different processing nodes that perform different corresponding functions. Each processing node is implemented using some portion of the computing resources provided by one or more computing devices. In some cases, a single computing device may implement a single processing node. In other cases, a single computing device may host two or more processing nodes; these processing nodes may perform the same function or any combination of different functions. For example, a single computing device may host two processing nodes that simultaneously perform function A and function B, or function A followed immediately by function B. In other cases, two or more computing devices may implement a single processing node. For simplicity, the following explanation will assume that each processing node is mapped to at least one computing device. However, based on the explanation above, this does not mean that a computing device is solely dedicated to implementing that processing node, or that the processing node is entirely hosted by the computing device.

[0040] The training framework 108 can generally be described as a functional flow that breaks down the training operation into iterative executions until the training objective is achieved. During each iteration, the training framework 108 also executes multiple instances of this flow. This mode of operation allows the training framework 108 to generate trained embeddings without overwhelming the processing power of any of its computing devices. This aspect of the training framework 108 also accelerates the generation of trained embeddings. As will be illustrated in more detail below, the training framework 108 can efficiently perform parallel operations because these operations are largely (or completely) independent of each other. For example, the operation depends on a minimum number of shared parameters (e.g., a single global bias parameter) or may have no shared parameters.

[0041] Referring now to the push-off phase environment 104, the similarity evaluation engine 120 uses trained embeddings provided in the data repository 116 to perform some type of real-time analysis or backend analysis. Figure 1 In the example, similarity assessment engine 120 receives a query submitted by a user via user computing device 122 through computer network 124. The similarity assessment engine 120 can then generate a similarity score, which indicates the degree to which the submitted query is considered similar to considered candidate options. For example, considered candidate options could correspond to documents, images, videos, etc. The similarity assessment engine 120 can then provide the user with output conveying the level of similarity.

[0042] In a business context, search engine 110 can combine the aforementioned type of similarity evaluation engine 120. That is, search engine 110 can use an initial matching process to identify a set of candidate options that may be suitable for the query. Then, search engine 110 can use the similarity evaluation engine 120 to continuously generate similarity scores for each candidate option in the set. Search engine 110 can then rank the candidate options based on their similarity scores and select the n candidates with the best similarity scores. Search engine 110 may optionally consider other matching criteria when generating this list, including any type of lexical features, semantic features, contextual features, etc. Finally, search engine 110 can generate a search results page that provides fragments associated with the top-ranked candidate options.

[0043] In another implementation, the similarity evaluation engine 120 can rely on any type of search algorithm to find the item embedding in the data repository 116 that is closest to the query embedding associated with the submitted query. For example, the similarity evaluation engine 120 can rely on the Approximate Nearest Neighbor (ANN) technique to perform this function. In this paper, the search engine 110 does not need to generate an initial list of candidates first in order to consider this.

[0044] Reference Figure 6The operation of the similarity evaluation engine 120 described above will be explained in more detail. As an introduction to the explanation below, the similarity evaluation engine 120 provides a query embedding based on one or more trained embeddings provided in the data repository 116. The similarity evaluation engine 120 also provides item embeddings for specific candidates under consideration based on one or more trained embeddings provided in the data repository 116. The similarity evaluation engine 120 can then determine the distance between the query embedding and the item embeddings in the vector space to provide a similarity score between the query and the candidates. It can use any metric such as cosine similarity, Manhattan distance, etc., to compute this distance.

[0045] The search applications described above are cited as examples, not as limitations. In another example, given a question submitted by a user, a chatbot engine can use the similarity evaluation engine 120 to determine the appropriate answer. In another example, given a user's current context, an advertising engine can use the similarity evaluation engine 120 to determine the appropriate advertisement to be offered to the user. For example, the current context may depend on the webpage the user is using and / or the click selection made by the user. In another example, an item organization engine can use the similarity evaluation engine 120 to generate clusters of embeddings that are within a specified distance from each other in a vector space. Candidate items associated with these mutually approximate embeddings form a set of similar items. For example, an image organization tool can use this feature to identify groups of related digital photos. In another example, a trend analysis engine can use the similarity evaluation engine 120 to determine trends in user behavior regarding the selection, editing, sending, and / or receiving of items.

[0046] In the following combination Figure 8 In another example, described in more detail, the application may use a similarity evaluation engine 120 to adjust clicks that are assumed to have been received by a first item. The application operates by distributing clicks received by the first item to at least one other relevant item. The number of clicks contributed by the first item to the other items depends on the degree of relation between the first item and that other item. The application determines the relationship between the two items by determining the distance between their embeddings in the vector space.

[0047] In some implementations, one or more servers 126 implement the complete similarity assessment engine 120. Users can interact with the similarity assessment engine 120 using a browser application provided by the user computing device 122. The user computing device 122 can correspond to any type of device performing computing functions, such as a desktop personal computing device, any type of handheld computing device (e.g., a smartphone, tablet computing device, etc.), a game console, an Internet of Things (IoT) device, a smart appliance, a wearable computing device, a mixed reality device, etc. The computer network 124 can correspond to a wide area network (e.g., the Internet), a local area network, one or more peer-to-peer communication links, etc., or any combination thereof. In other implementations, each user computing device can provide a local representation of the similarity assessment engine 120. In other implementations, the server 126 implements some functions of the similarity assessment engine 120, and each local user computing device 122 implements other functions of the similarity assessment engine 120.

[0048] As a first general point, the similarity evaluation engine 120 relies on trained embeddings already generated by the training framework 108. Because of this, the similarity evaluation engine 120 does not necessarily need to execute complex machine-trained models when processing user queries. Therefore, compared to systems that invoke complex machine-trained models to process queries, the similarity evaluation engine 120 can be expected to consume fewer computational resources. Computational resources include memory resources, processing resources, and data storage resources. The similarity evaluation engine 120 can also be expected to provide better latency-related performance compared to systems that invoke complex machine-trained models to process user queries.

[0049] In fact, whenever a new training sample is introduced, the training phase environment 102 must re-invoke the training framework 108. This is appropriate, for example, when developers want to add new item embeddings to the data repository 116. However, the cost of this operation is preferable for dynamically generating embeddings using a machine-trained model during inference phase processing. It should also be noted that new training samples typically require generating and adding only a small number of new embeddings to the data repository 116, rather than readjusting all the weights of a large machine-trained model. The training phase environment 102 achieves this behavior because the work it performs on embeddings can be largely or entirely separated from the work it performs on other embeddings. Furthermore, the focus of training is on the generation of specific embeddings, rather than generating or updating the massive machine-trained model that will be deployed in the inference phase environment 104. Another way to achieve this is that the training phase environment 102 is scalable, unlike traditional machine training platforms.

[0050] As a second general point, the similarity evaluation engine 120 can operate based on relatively small-sized embeddings, for example, in a purely illustrative case with a dimension of k=16. This factor also makes the similarity evaluation engine 120 more resource-efficient compared to engines that involve processing larger vectors.

[0051] As a third general point related to the first, the training framework 108 uses a first prediction function in generating the trained embeddings. That is, the training framework 108 uses the first prediction function to generate predictions indicating whether a query specified in the training samples will lead to a click on an item also specified in the training samples. The similarity evaluation engine 120 uses a second prediction function that measures the degree to which a submitted query is considered similar to a candidate item. The first prediction function does not necessarily have to be the same as the second prediction function. In fact, the second prediction function can be less complex and less computationally intensive than the first prediction function. This aspect of the inference phase environment 104 also separates it from traditional machine training models. Typically, the machine training model applied in the inference phase represents the same model that has been trained in the training phase. Figure 1 The method used is efficient because it involves using a single set of embeddings for multiple potential downstream applications, each using different logic to evaluate similarity. In other words, the method is efficient because it eliminates the time and resources required to generate custom embeddings specifically for particular end-use applications.

[0052] Figure 3 An overview of different types of processing nodes that can be used by the training framework 108 is provided. A first group of multiple processing nodes 302 generates initial embeddings for the training sample set. These types of processing nodes are referred to herein as embedding initialization nodes. Two or more computing devices implement embedding initialization nodes. A second group of multiple processing nodes 304 performs corresponding local training operations on the corresponding samples to produce local lexical embeddings. These types of processing nodes are referred to herein as embedding update nodes. Two or more computing devices implement embedding update nodes. A third group of multiple processing nodes 306 merges multiple groups of local lexical embeddings to produce merged lexical embeddings. These types of processing nodes are referred to herein as embedding merging nodes. Two or more computing devices implement embedding merging nodes. Note that the above node types are examples and not restrictive references; other implementations may divide the training task into other combinations of node types. It should also be noted that any computing device can implement two or more different types of processing nodes.

[0053] In one implementation, the training framework 108 includes a central control computing device (not shown) that initiates the training process and performs other high-level configuration tasks. Otherwise, the processing nodes in the training framework 108 operate autonomously, sending information to each other in a peer-to-peer manner.

[0054] Figure 4 It shows Figure 4 An example of the operation of training framework 108. In this simplified case, the training samples include a first sample where at least one user has submitted the query "funny dog ​​video" and then clicked item D3. The training samples include a second training sample where at least one user has submitted the query "new cat video" and then clicked item D2. The training samples include a third sample where at least one user has submitted the query "funny video of cat" and then clicked item D1. Figure 4 In the example, training framework 108 processes queries at the word-level granularity. However, other implementations can process queries at the n-gram level (such as ternary grammar granularity) or a combination of n-gram and word-level granularity. Although in Figure 4 The example is not shown, but the training framework 108 also processes negative training samples. As described above, the sample mining system 106 generates negative training samples by randomly associating a query with an item based on the assumption that the item does not match the query.

[0055] In the first operation 402, the training framework 108 uses one or more embedding initialization nodes (A1, A2, A3, A4) to generate initial embeddings for unique lexical units in the training samples. For example, embedding initialization node A1 generates initial embeddings for the words "funny" and "new". Embedding initialization node A2 generates initial embeddings for words such as "cat" and "dog". The training framework 108 can use any mapping function to assign words to the embedding initialization nodes. For example, the training framework 108 can hash each word to a key and then use the key to find the embedding initialization node that will process that word. Each embedding initialization node can use any function to generate initial embeddings, such as using a trained neural network or lookup table to map words to vectors.

[0056] In operation 404, training framework 108 may assign the training sample set to one or more embedding update nodes (B1, B2, B3). For example, training framework 108 may assign a third training sample to embedding update node B1, a second training sample to embedding update node B2, and a first training sample to embedding update node B3. Training framework 108 may use any mapping function to assign words to embedding initialization nodes. For example, training framework 108 may hash each query and item identifier associated with a training sample to a key, and then use the key to find the embedding update node that will process the training sample.

[0057] In operation 404, the training framework 108 also distributes the initial embeddings to the selected embedding update nodes. For example, the training framework 108 distributes the initial embeddings of "funny", "video", "of", and "cat" to the embedding update node B1, because this node will need these embeddings to process the third sample that has been assigned to it.

[0058] In operation 406, each embedding update node generates a prediction of whether a query associated with the training samples will result in a click on an item associated with the training samples. For example, embedding update node B1 will apply a prediction function to determine whether the specified query "funny video of cat" will result in a click on item D1. The training samples also provide ground truth information indicating whether an item actually matches a query. In the case of embedding update node B1, it is assumed that item D3 is indeed a good match for the query "funny video of cat".

[0059] Then, each embedding update node updates the embedding associated with the training sample under consideration based on how consistent or inconsistent its predictions are with the ground truth. This task can be performed using different methods, such as stochastic gradient descent (SGD), or optionally momentum. In one implementation, the embedding update node uses logistic loss during its training, although other implementations can use other types of loss functions. Once training is complete, the momentum-related parameter values ​​(if used) are discarded and are not used in the inference phase environment 104.

[0060] For example, suppose that prediction indicator D1 generated by embedding update node B1 is not a good match for “funny video of cat”, even though it is known that it is actually a good match. In response, embedding update node B1 will adjust the embeddings associated with the training samples to penalize its inaccurate prediction. If prediction indicator D1 is a good match, embedding update node B1 will adjust the embeddings associated with the training samples to also support the prediction it made. In the terminology used herein, each embedding update node produces a local set of lexical embeddings. For example, embedding update node B1 produces an updated set of embeddings for the lexicals “funny”, “video”, “of”, “cat”, and “D1”. The embeddings are called “local” because they reflect insights gathered from predictions made solely by embedding update node B1 and do not reflect insights gathered from other embedding update nodes (e.g., B2 and B3) operating in parallel with embedding update node B1. Predictions made by embedding update node B1 also do not depend on predictions made by other embedding update nodes, or do not require communication with these other embedding update nodes. This is true even when different embedding update nodes are implemented by the same computing device, and even when different embedding update nodes process queries that include at least one common terminology (e.g., the word "funny").

[0061] In operation 408, the training framework 108 distributes local lexical embeddings to appropriate embedding merging nodes (C1, C2, C3, C4). The training framework 108 can perform this task as described above, for example, by hashing each lexical to generate a key, and then using the key to send the local lexical embedding associated with that lexical to the appropriate embedding merging node. Note that while conventional machine training platforms operate by adjusting the gradient for each iteration, the training framework 108 actually moves the local embeddings between separate processing nodes.

[0062] In operation 410, each embedding merging node merges one or more examples of local lexical embeddings for each lexical it processes. For example, embedding merging node C1 receives local lexical embeddings for the word "funny" from embedding update node B1. Embedding merging node C1 also receives local lexical embeddings for the same word "funny" from embedding update node B3. The two vectors are then averaged on a dimension-by-dimensional basis to produce the merged lexical embedding for the lexical "embedding".

[0063] In operation 412, the training framework 108 distributes the merged lexical embeddings generated in operation 410 to the embedding update nodes that will need them to perform subsequent operations. For example, suppose embedding update node B1 will next process another training sample with a query that includes the word "funny". In response, the training framework 108 sends the merged lexical embeddings for the word "funny" to the embedding update node B1. When performing the next update operation, the embedding update node B1 will rely on the merged lexical embeddings for the word "funny", as well as other merged lexical embeddings received from one or more other embedding merge nodes. Operation 412 also means that embedding update nodes can host different subsets of the complete set of embeddings. For example, different embedding update nodes can host different embeddings associated with different subsets of words within the entire vocabulary of the word.

[0064] In operation 414, the training framework 108 repeats operations 406 to 412 one or more times until the training objective is achieved. For example, the training framework 108 may repeat operations 406 to 412 until the trained embedding produces a prediction accuracy level that meets a specified threshold.

[0065] Figure 5 It shows in Figure 1 and Figure 3 This is an example of the processing logic used in the illustrative computing device 502 within the training framework 108. In this case, it is assumed that the single computing device 502 can play a role associated with the embedding initialization node, the embedding update node, and the embedding merging node. In other implementations, the computing device may execute a subset, but not all, of the functions supported by the training framework 108.

[0066] Initialization logic 504 generates an initial embedding for each word assigned to computing device 502. Initialization logic 504 can perform this task by using the same predetermined initial embedding for all words. Alternatively, initialization logic 504 can map the received words to the initial embedding using any mapping function such as a pre-trained neural network, a lookup table, etc.

[0067] Prediction logic 506 generates predictions for a given training sample x, which includes a given query and a given item identifier. In one implementation, prediction logic 506 relies on the following training-stage prediction function to generate predictions.

[0068]

[0069] In this formula, x iThis refers to each word element that constitutes the training sample x. Word elements include query words that constitute the query and individual item words that provide item identifiers. The symbol w0 is the global bias parameter, and each w... i It is applied to each word x considered in the training sample x. i The weighted parameters are referred to as local bias parameters. The global bias parameter w0 is a shared parameter that can be optionally omitted during training to eliminate dependencies between parallel processes performed by the nodes of the distributed training framework 108. More specifically, the global bias parameter can be omitted because this omission does not adversely affect the quality of the trained embeddings, and at least in some inference stage engines using the trained embeddings, the global bias parameter is not used. Eliminating the global bias parameter during training provides an efficiency improvement because if the global bias parameter is included, the global bias parameter value must be averaged over every sample in the dataset for each iteration. However, in other implementations, the global bias parameter is included in training in cases where downstream inference stage tasks rely on the global bias parameter.

[0070] symbol V i Reference and lexical x i The embedding of the associated size k. Symbol v j Referent and word element x j Another embedding of the associated size k. Item <v i v j >Refers to embedded V i and v j The dot product, embedded in V i and v j With word element x i and word element x j Related, for example In general, Equation (1) can be summarized as mapping the embedding vector associated with a sample to a predicted function F, which indicates whether a query specified in the training samples is a good match for an item identifier specified in the training samples. Although not expressed by Equation (1), prediction logic 506 may optionally pass the output of Equation (1) through an activation function, such as a sigmoid function (which is suitable for binary classification).

[0071] Despite Figure 3 and 4 The description does not mention this, but the embedding update nodes also calculate the local bias parameter values ​​(w) of the tokens they process. iThe embeddings are then sent to the appropriate embedding merging node. The embedding merging node can then average the values ​​of these local bias parameters for a specific lemma across multiple examples received by each embedding merging node to produce a merged local bias parameter value. This merged local bias parameter value can then be distributed to selected embedding update nodes. For example, suppose two embedding update nodes process the word "funny". They will produce two local versions of the embeddings for that word and two local versions of the local bias parameter values ​​for that word. They will send this information to the same embedding merging node. The embedding merging node will then average the two local embeddings for "funny" to produce a merged embedding for "funny", and average the two local bias parameter values ​​for "funny" to produce a merged local bias parameter value for "funny". The embedding merging node will then send this merged information to any embedding update nodes that need it to perform update operations in subsequent training loops.

[0072] However, in other implementations, the local bias parameter values ​​are not used during the training and inference phases, and therefore the second term of equation (1) can be omitted entirely, as can the associated processing. Even if included, the inference phase engine does not need to utilize the local bias parameter values ​​(although it is optional).

[0073] The local update logic 508 uses stochastic gradient descent or some other iterative training algorithm to update the embeddings to produce local lexical embeddings. In one implementation, the local update logic 508 can perform this operation using the following formula:

[0074]

[0075] Synchronization logic 510 uses any merging function to merge two or more embeddings of a word. In one case, synchronization logic 510 merges embeddings by averaging them on an element-by-element basis. For example, if there are three embeddings for the word "video", synchronization logic 510 generates the first element of the merged embedding by adding the three values ​​in the first element of the three vectors and dividing by three.

[0076] Distribution logic 512 sends the output generated by the computing device to one or more downstream components. In one context, the output corresponds to one or more initial embeddings. In another context, the output corresponds to a local lexical embedding set. In yet another case, the output corresponds to one or more global lexical embeddings.

[0077] Figure 6 A similarity evaluation engine 602 is shown, which represents the above combination Figure 1This description introduces an implementation of the similarity evaluation engine 120 during the inference phase. It is assumed that the similarity evaluation engine 602 is currently performing the task of determining whether a candidate is a good match for the submitted query. The similarity evaluation engine 602 may receive the query from the user's computing device 122 or some other mechanism. The similarity evaluation engine 602 may receive item identifiers identifying the candidates from the search engine 110 or any other context-specific query processing function. The embedding receiving logic 604 then retrieves a trained embedding from the data repository 116 associated with this input information, where such a trained embedding has previously been generated by the training framework 108. More specifically, the embedding receiving logic 604 may retrieve the query component embedding Q associated with each terminology (e.g., each word) of the query. i The embedded receiving logic 604 can also retrieve the item embedding d associated with the item identifier.

[0078] Similarity evaluation logic 606 generates a similarity score based on the trained embeddings described above. For example, summation logic 608 can first be based on the embeddings of all |Q| query components in the input query. i To generate a single query embedding q.

[0079]

[0080] The distance calculation logic 610 then uses cosine similarity or any other distance metric to determine the similarity score s. For cosine similarity, the similarity score s is given by the following formula:

[0081]

[0082] The training framework 108 generates the item embedding d used in equation (4) by operating on training samples that include item identifiers d associated with specific items. As explained above, the item identifier can correspond to the URL associated with the item. In other words, the training framework 108 treats the item identifier d itself as a lexical without considering the content of the item.

[0083] In summary, note that the training framework uses Equation (1) to perform predictions, while the similarity evaluation engine 120 uses Equations (3) and (4) to perform predictions. This provides a concrete example of how the similarity evaluation engine can fully utilize the trained embeddings generated by the training framework 108, but does not need to use the same prediction function as the training framework 108. In other words, the prediction function applied by the training framework 108 can be considered as a proxy for one or more other prediction functions applied by the push-out phase.

[0084] Figure 7The operation of the similarity assessment engine 602 relative to a set of vectors in a vector space is illustrated. (Note that the vector space is shown only in two dimensions for ease of interpretation; the vector space actually has k dimensions.) The query embedding q in the vector space consists of query component embeddings Q associated with the terms constituting the query. i The position is determined by the sum of the terms. Therefore, it can be expected that the query embedding q is located in the position determined by the query component embedding Q. i A specific point within a defined cluster of points. Distance calculation logic 610 determines the embedding d of each item. i The distance to the query embedding. The item embedding located closest to the query embedding q can be expected to be most relevant to the query embedding q.

[0085] In another implementation, the summation logic 608 can use equation (3) to embed the header calculation item d, i.e., by using Q i Replace with T i T i The item component embeddings are referenced to the terms (e.g., words) that appear in the item title. Alternatively or additionally, the summation logic 608 may add the embeddings associated with the terms to some other identified parts of the item, such as its summary or keyword section. In one implementation, the training framework 108 still operates in the same way as described above, regardless of the content of any item. In other words, the training framework 108 performs analysis on the identifiers associated with the item, rather than its internal terms.

[0086] In the third case, the similarity assessment engine 120 provides a hybrid model. If the data repository 116 includes item embeddings for candidates under consideration, the hybrid model applies equations (3) and (4), where again, the item embedding represents a transformation of the item's identifier rather than its internal content. However, if item embeddings are not available in the data repository 116, the hybrid model generates a synthesized item embedding in the manner described above by summing up item component embeddings d associated with lemmas in the item title or some other part. When using the second option, the hybrid model can apply a discount factor to the similarity score, which has the effect of reducing the similarity score. This reflects the fact that similarity scores generated based on words in the item title may be less reliable than similarity scores generated based on item embeddings included in the data repository 116. This is because each item embedded in the data repository 116 is derived from evidence about user click behavior for a particular item.

[0087] Figure 8 A similarity evaluation engine 802 is shown, which indicates that in Figure 1 This is another implementation of the similarity evaluation engine 120 described in the context of the description. The goal of the similarity evaluation engine 802 is to evaluate similarity evaluation engines based on at least one other item (d) relative to submissions of the same query q.i Adjust the target item d based on the number of clicks already received. j The number of clicks received. This is determined by embedding V with the identifier associated with the target item. j The V-shaped embedding identifies items associated with other items. j Then, for example, cosine similarity or any other distance metric is used to determine the two embeddings ((v j v i The relationship between )) in the vector space is used to determine other terms d. i "Contribute" to target item d j The number of clicks.

[0088] More specifically, Figure 8 An example is shown where table 804 or another data structure in memory stores a list of queries submitted by the user and information about the number of times the user clicked on those items. For example, the first row 806 of the table indicates that the user clicked the first item D1 a total of 325 times after submitting query Q1. The user clicked the second item D2 only 3 times when submitting the same query Q1. Note that the actual table can present this click information in any form, such as normalized click-through rate (CTR) information; Figure 8 An example is shown that only the raw click information is provided for ease of interpretation.

[0089] The difference in the number of clicks received by items D1 and D2 can indicate some contextual issues. For example, consider a scenario where search engine 110 presents a search results page with a list of matching items when a user enters query Q1. Also assume that search engine 110 displays the entry for item D1 at the top of that list because it is determined to be the most relevant item to the user's query Q1. After receiving this search results page, most users can be expected to click on the entry for item D1. Assuming that the item satisfies their search goal, it is also expected that most users will terminate their search at this stage. This prevents users from discovering that the entry for item D2, which appears further down in the list, is also relevant to their search goal. Search engine 110 is unlikely to elevate item D2's position in the list over time because few users will click on it. On the other hand, search engine 110 will likely reinforce its conclusion that item D1 is relevant because many users continue to click on it. This has the effect of effectively trapping item D2 in a disadvantageous position within the list, even if it provides information as valuable as item D1. The similarity assessment engine 602 operates in this context to increase the number of clicks that item D2 is considered to have received when reaching the conclusion that item D2 is sufficiently relevant to item D1, and thus may also satisfy the user's search objective. By increasing the number of clicks received by item D2, the search engine 110 can respond by increasing its appearance on the search results page. This behavior helps users discover relevant items.

[0090] More specifically, the embedding receiving logic 808 receives input information from table 804. Relative to the first row 806, the input information includes information identifying query Q1, the item (D1, D2, D3, ...) selected by the user in response to query Q1, and raw click information. The click information is referred to as "raw" because it reflects the actual selection made by the user. The embedding receiving logic 808 then retrieves the embedding associated with each of query Q1 and item (D1, D2, D3, ...). More specifically, in one implementation, the embedding receiving logic 808 retrieves the same type of embeddings as described above generated by the training framework 108. Recall that the training framework 108 generates embeddings by operating on a corpus of training samples that include paired query and item identifiers. Item identifiers may correspond to URLs. Additionally or alternatively, the training framework 108 can generate item embeddings by identifying the set of lexical units associated with an item (such as the words that make up its title), identifying the item component embeddings associated with those lexical units, and summing those item component embeddings in the manner described above.

[0091] In other implementations, the embedding receiving logic 808 retrieves embeddings provided by sources other than the training framework 108. For example, the embedding receiving logic 808 may retrieve embeddings pre-generated by any type of deep neural network (DNN) such as a convolutional neural network (CNN). In some cases, such a DNN may map terms associated with an item to the embedding. In yet another case, the embedding receiving logic 808 relies on an algorithm that dynamically computes the embeddings upon request, i.e., rather than pre-compiling and storing the embeddings.

[0092] Click adjustment logic 810 adjusts the click information in the first row 806 of the original table 804 to produce the modified row 806' of the modified table 804'. In a non-restrictive method, click adjustment logic 810 uses the following formula to adjust the target item d in the first row 806. j Number of clicks received:

[0093]

[0094] symbol Referent d j The number of clicks received, where the superscript "c" indicates that these are conditional, proxy, or dummy clicks. Refers to another item d in line 806 of the first line. i The actual number of clicks received, where the superscript "o" indicates these clicks as either actual or original clicks. Item f(v) j v i ) refers to the embedding (v) that associates the measurement with the two terms under consideration. j v iThe similarity between (5) and (6) is any function. In a non-restrictive case, this function may correspond to cosine similarity. Although not shown, the click adjustment logic 810 may normalize the click information generated by equation (5) in any way, such as by dividing the click information by the average number of clicks in row 806 or the entire table 804.

[0095] Consider applying the above formula to the number of clicks received by item D2 in the first row 806. Click adjustment logic 810 can determine the relationship between item D2 and each other item in the first row 806. Click adjustment logic 810 boosts the click count of item D2 by a weighted sum of the click counts associated with other items, where the weight associated with each contribution is determined by the function f(v j v i Confirmed. Figure 8 In this illustrative example only, this has the net effect of increasing the click count for item D2 from 3 clicks to 63 clicks.

[0096] The optional downstream ranking engine 812 may rank items or perform any other processing on items based on the modified counting information in Table 804' and / or on any other features associated with the item and the context of the accessed item (such as lexical features, semantic features, contextual features, etc.).

[0097] Figure 9 The above scenario is illustrated relative to the placement of the embeddings in the vector space. The distance between each item embedding and the embedding of item D2 affects the relative number of clicks it will contribute to the modified click count of item D2. For example, the embedding associated with item D1 is closest to the embedding associated with item D2. Furthermore, item D1 has a relatively large click count compared to the other items in the first row 806. These two factors make it possible for item D1 to contribute the largest number of click counts to item D2. On the other hand, when repeating equation (5) for the case where item D1 is the target item, it can be expected that the click adjustment logic 810 will reduce the number of clicks considered to have been received by item D1. Overall, the click adjustment logic 810 has the effect of balancing the number of clicks received by closely related sets of phases, based on the premise that they are similar and therefore should have similar click counts.

[0098] B. Exemplary Process

[0099] Figures 10 to 12The operation of the training phase environment 102 and the inference phase environment 104 in Explanation Section A is illustrated in flowchart form. Since the basic principles of the operation of these environments (102, 104) have already been described in Section A, some operations will be explained in an overview manner in this section. As noted in the preamble to the detailed implementation, each flowchart represents a series of operations performed in a specific order. However, the order of these operations is merely representative and can vary in any way.

[0100] More specifically, Figure 10 It shows the provision Figure 1 The process 1002 outlines an operational mode of the training phase environment 102. In box 1004, the training phase environment 102 provides a set of training samples, each of which describes a query submitted by at least one user, an item, and an indication of whether that item has been selected by at least one user in response to submitting the query. In box 1006, the training phase environment 102 provides initial lexical embeddings associated with the different training samples to multiple corresponding embedding update computing devices. The training phase environment 102 then begins a loop, which is repeated multiple times until the training objective is achieved. In the looping box 1008, the training phase environment 102 uses multiple embedding update computing devices 304 to generate multiple sets of local lexical embeddings. In the looping box 1010, the training phase environment 102 provides multiple sets of local lexical embeddings to multiple embedding merging computing devices 306. In loop 1012, the training phase environment 102 uses multiple embedding merging computation devices to generate multiple merged lexical embeddings from multiple sets of local lexical embeddings; each merged lexical embedding for a particular lexical represents a merge of one or more local lexical embeddings for that particular lexical. In loop 1014, the training phase environment 102 provides the multiple merged lexical embeddings to a selected embedding update computation device. Operation 1016 instructs the loop to repeat at least once until the training objective is reached.

[0101] Figure 11 It shows the provision Figure 1The process 1102 outlines one mode of operation of the similarity evaluation engine 120. In box 1104, the similarity evaluation engine 120 receives identifiers associated with a first and a second item to be compared. In box 1106, the similarity evaluation engine 120 identifies a first embedding associated with the first item. In box 1108, the similarity evaluation engine 120 identifies a second embedding associated with the second item. In box 1110, the similarity evaluation engine 120 determines the relationship between the first and second items by determining the relationship between the first and second embeddings in the vector space to provide a similarity score. In box 1112, the similarity evaluation engine provides an output result dependent on the similarity score. The operations of identifying the first and second embeddings in boxes 1106 and 1108 utilize a data repository 116 of trained embeddings generated by a training framework 108. The training framework 108 uses a distributed collection of computing devices 118 and uses a prediction function to generate trained embeddings that differs from the prediction function used in the operation of determining the relationship between the first and second items.

[0102] exist Figure 6 In the context of the similarity assessment engine 602 shown, the first item mentioned above is a query submitted by a user via user computing device 122. The second item is a candidate item under consideration. The identification operation in box 1106 involves identifying query embeddings associated with one or more terms of the query. The identification operation in box 1108 involves identifying item embeddings associated with candidate items. The determination operation in box 1110 involves determining the distance between the query embedding and the item embedding, which corresponds to a similarity score. The operation provided in box 1112 involves sending search results identifying the relationship between candidate items and the query to user computing device 112.

[0103] exist Figure 8 In the context of the similarity assessment engine 802, the first item is the item selected by at least one user for a first number of times, as reflected by the first original count information. The second item is the item selected by at least one user for a second number of times, as reflected by the second original count information, where the second number differs from the first number. Process 1102 also involves adjusting the count information for the second item by weighting the count information for the first item with a similarity score, to provide revised count information.

[0104] Figure 12 Description is shown Figure 8 A process 1202 describes one way of operating a specific similarity assessment engine 802. This process 1202 has the effect of adjusting the number of clicks received by one or more documents. In box 1204, the similarity assessment engine 802 accesses data structures, such as... Figure 4Table 804 is shown. In box 1206, the similarity evaluation engine 802 receives raw count information for the first item from the data structure, indicating the first item has been selected for the first time in response to a query submission. In box 1208, the similarity evaluation engine 802 receives additional instances of raw count information for multiple other items from the data structure, each instance of the additional raw count information indicating that another item has been selected multiple times in response to the same query submission. In box 1210, the similarity evaluation engine 802 identifies a first embedding associated with the first item. In box 1212, the similarity evaluation engine 802 identifies other embeddings associated with other items respectively. In box 1214, the similarity evaluation engine 802 determines the relationship between the first item and each of the other embeddings by determining the relationship between the first embedding in the vector space and each of the other embeddings to provide multiple similarity scores. In box 1216, the similarity evaluation engine 802 adjusts the raw count information of the second item by additional examples of raw count information weighted by the corresponding similarity scores to provide revised count information. Then, the similarity evaluation engine 802 stores the adjusted count information in a data structure.

[0105] C. Representative calculation function

[0106] Figure 13 A computing device 1302 is shown that can be used to implement any aspect of the mechanisms illustrated in the above figures. For example, refer to Figure 1 , Figure 13 The type of computing device 1302 shown can be used to implement any user computing device or any server. In all cases, computing device 1302 represents a physical and tangible processing mechanism.

[0107] The computing device 1302 may include one or more hardware processors 1304. The hardware processors 1304 may include, but are not limited to, one or more central processing units (CPUs), and / or one or more graphics processing units (GPUs), and / or one or more application-specific integrated circuits (ASICs). More generally, any hardware processor may correspond to a general-purpose processing unit or a dedicated processing unit.

[0108] The computing device 1302 may also include a computer-readable storage medium 1306 corresponding to one or more computer-readable medium hardware units. The computer-readable storage medium 1306 retains information 1308 of any type, such as machine-readable instructions, settings, data, etc. For example, without limitation, the computer-readable storage medium 1306 may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tapes, etc. Any instance of the computer-readable storage medium 1306 may use any technology for storing and retrieving information. Furthermore, any instance of the computer-readable storage medium 1306 may represent a fixed or removable unit of the computing device 1302. Additionally, any instance of the computer-readable storage medium 1306 may provide volatile or non-volatile retention of information.

[0109] The computing device 1302 may utilize any instance of the computer-readable storage medium 1306 in different ways. For example, any instance of the computer-readable storage medium 1306 may represent a hardware storage unit (such as random access memory (RAM)) for storing transient information during program execution on the computing device 1302, and / or a hardware storage unit (such as a hard disk) for retaining / archiving information on a more permanent basis. In the latter case, the computing device 1302 may also include one or more drive mechanisms 1310 (such as hard disk drive mechanisms) for storing and retrieving information from instances of the computer-readable storage medium 1306.

[0110] The computing device 1302 may perform any of the functions described above when the hardware processor(s) 1304 executes computer-readable instructions stored in any instance of the computer-readable storage medium 1306. For example, the computing device 1302 may execute computer-readable instructions to perform each block of the process described in Part B.

[0111] Alternatively or additionally, computing device 1302 may rely on one or more other hardware logic units 1312 to perform operations using a task-specific set of logic gates. For example, the hardware logic unit(s) 1312 may include a fixed configuration of hardware logic gates, created and set at manufacturing time and not subsequently changed. Alternatively or additionally, the other hardware logic unit(s) 1312 may include a set of programmable hardware logic gates that can be configured to perform different dedicated tasks. This latter type of device includes, but is not limited to, programmable array logic devices (PALs), general-purpose array logic devices (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), etc.

[0112] Figure 13Generally, hardware logic circuitry 1314 includes any combination of hardware processor(s) 1304, computer-readable storage medium(s) 1306, and / or other hardware logic units(s) 1312. That is, computing device 1302 can employ any combination of hardware processor(s) 1304 that executes machine-readable instructions provided in computer-readable storage medium(s) 1306 and / or one or more other hardware logic units(s) 1312 that perform operations using a fixed and / or programmable set of hardware logic gates. More generally, hardware logic circuitry 1314 is associated with one or more hardware logic units(s) of any type that perform operations based on logic stored in and / or otherwise included in the hardware logic units(s).

[0113] In some cases (e.g., where computing device 1302 represents a user computing device), computing device 1302 also includes an input / output interface 1316 for receiving various inputs (via input device 1318) and providing various outputs (via output device 1320). Illustrative input devices include keyboard devices, mouse input devices, touchscreen input devices, digitizers, one or more still image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, voice identification mechanisms, any motion detection mechanisms (e.g., accelerometers, gyroscopes, etc.), etc. A particular output mechanism may include display device 1322 and an associated graphical user interface (GUI) presentation 1324. Display device 1322 may correspond to liquid crystal display devices, light-emitting diode display (LED) devices, cathode ray tube devices, projection mechanisms, etc. Other output devices include printers, one or more speakers, haptic output mechanisms, archiving mechanisms (for storing output information), etc. Computing device 1302 may also include one or more network interfaces 1326 for exchanging data with other devices via one or more communication channels 1328. One or more communication buses 1330 communicatively couple the above units together.

[0114] The multiple communication channels 1328 can be implemented in any way, such as through a local area computer network, a wide area computer network (e.g., the Internet), a point-to-point connection, or any combination thereof. The multiple communication channels 1328 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., managed by any protocol or combination of protocols.

[0115] Figure 13 A computing device 1302 consisting of a discrete set of separate units is shown. In some cases, the set of units may correspond to separate hardware units provided in a computing device chassis of any shape. Figure 13An exemplary shape element is shown in its bottom portion. In other cases, computing device 1302 may include integrated... Figure 1 The hardware logic unit that provides the functionality of two or more units as shown. For example, computing device 1302 may include a system-on-a-chip (SoC or SOC) that combines... Figure 13 The integrated circuits shown correspond to the functions of two or more units.

[0116] The following overview provides a non-exhaustive collection of illustrative examples of the techniques described in this article.

[0117] According to a first example, a computer-implemented method for generating a trained set of embeddings is described. The method includes providing a set of training samples, each training sample describing a query submitted by at least one user, an item, and an indication of whether the item has been selected by at least one user in response to submitting the query. The method also includes providing initial lexical embeddings associated with the different training samples to a plurality of corresponding embedding-updating computing devices. The method then performs an iterative training loop, including: generating a plurality of local lexical embedding sets using the plurality of embedding-updating computing devices; providing the plurality of local lexical embedding sets to a plurality of embedding-merging computing devices; generating a plurality of merged lexical embeddings from the plurality of local lexical embedding sets using the plurality of embedding-merging computing devices, each merged lexical embedding for a particular lexical represents a merge of one or more local lexical embeddings for the particular lexical; and providing the plurality of merged lexical embeddings to a selected embedding-updating computing device. The method also includes repeating the training loop until a training objective is achieved, at which point the plurality of merged lexical embeddings provided to the embedding-updating computing devices correspond to the trained set of embeddings.

[0118] According to the second example, at least one of the multiple embedded update computing devices is the same computing device as the embedded merge computing device.

[0119] According to the third example, each particular training sample uses one or more query terms to represent a particular query, where each query term is represented as a query term vector with a dimension as large as the number of query terms in the vocabulary of the query terms.

[0120] According to the fourth example, each specific training sample is represented by a term vector with a dimension as large as the number of terms in the set of options.

[0121] According to the fifth example, each embedding update computing device generates a local lexical embedding set by the following steps: for specific training samples identifying a specific query and a specific item, generating a prediction indicating whether the specific query is considered similar to the specific item; and updating the local lexical embedding set based on an indication that the prediction is consistent with a specific label associated with the specific training sample. Each embedding update computing device performs a process that relies on no more than one global parameter value shared with processes performed by other embedding update computing devices.

[0122] According to the sixth example, a specific merged lexical embedding for a particular lexical is generated by averaging two or more local lexical embeddings for the particular lexical together.

[0123] According to the seventh example, the operation of providing multiple merged lexical embeddings to a selected embedding update computing device involves sending a given merged lexical embedding associated with the given lexical to each embedding update computing device that has been assigned at least one training sample including the given lexical.

[0124] According to the eighth example relating to the seventh example, an embedding update computing device that has not been assigned at least one training sample including a given lexical does not receive a given merged lexical embedding.

[0125] According to Example 9, a trained embedding is used during the inference phase to determine a similarity score that measures the level of similarity between a user-submitted query and its candidates.

[0126] According to the tenth example, trained embeddings are used during the inference phase, and the inference phase uses a prediction function to generate multiple sets of local lexical embeddings, which is different from the prediction function used in the training loop.

[0127] According to Example 11, a computer-implemented method for determining a relationship between two or more items is described. The method includes: receiving identifiers associated with a first item and a second item to be compared; identifying a first embedding associated with the first item; identifying a second embedding associated with the second item; determining the relationship between the first and second items by determining the relationship between the first and second embeddings in a vector space to provide a similarity score; and providing an output result dependent on the similarity score. The operations of identifying the first and second embeddings utilize a data repository of trained embeddings generated by a training framework. The training framework uses a distributed collection of computing devices and employs a prediction function different from the prediction function used in the operation of determining the relationship between the first and second items.

[0128] According to Example Twelve, which relates to Example Eleven, the first item is a query submitted by a user via a user's computing device. The second item is a candidate item under consideration. Identifying the first embedding involves identifying a query embedding associated with one or more terms of the query. Identifying the second embedding involves identifying a term embedding associated with a candidate item. Determining the relationship involves determining the distance between the query embedding and the term embedding, which corresponds to a similarity score. Providing output results includes sending search results identifying the relationship between the candidate item and the query to the user's computing device.

[0129] According to the thirteenth example involving the twelfth example, the operation of identifying query embeddings includes: identifying multiple query component embeddings associated with query terms of the query, respectively; and summing the multiple query component embeddings to produce a query embedding.

[0130] According to the fourteenth example involving the twelfth example, the operation of identifying item embedding includes retrieving a single item embedding from a data repository associated with the item identifier of the identifying candidate.

[0131] According to the fifteenth example involving the twelfth example, the operation of identifying item embeddings includes: identifying multiple item component embeddings that are associated with at least a portion of the lexical units forming the candidate item; and summing the multiple item component embeddings to produce an item embedding.

[0132] According to the sixteenth example involving the fifteenth example, a term embedding is generated by summing multiple term component embeddings when it is determined that no single term embedding exists in the data repository associated with the term identifier of the term candidate.

[0133] According to the seventeenth example involving the fifteenth, multiple item components embed corresponding words in the title that describe the candidate.

[0134] According to the eighteenth example relating to the eleventh example, the first item is an item in which at least one user selects a first number, as described by the first original count information associated with the first item. The second item is an item in which at least one user selects a second number, as reflected by the second original count information associated with the second item, the second number being different from the first number. The computer-implemented method further includes generating modified count information for the second item based on the first original count information weighted by a similarity score. The method includes storing the modified count information.

[0135] According to the nineteenth example involving the eighteenth example, the computer-implemented method also includes adjusting the ranking of the second item relative to the first item in the search results provided to the user based on the modified counting information.

[0136] According to the twentieth example, a computer-readable storage medium for storing computer-readable instructions is described. The computer-readable instructions, when executed by one or more hardware processors, perform a method comprising: accessing a data structure; receiving from the data structure raw count information of a first item, the raw count information indicating that the first item has been selected for the first time in response to a query submission; receiving from the data structure other instances of raw count information of a plurality of other items, each instance of the other raw count information indicating that another item has been selected multiple times in response to the same query submission; identifying a first embedding associated with the first item; identifying other embeddings associated with other items respectively; determining a relationship between the first item and each of the other items by determining a relationship between the first embedding and each of the other embeddings in a vector space to provide a plurality of similarity scores; adjusting the raw count information for a second item by other instances of the raw count information weighted by the corresponding similarity scores to provide corrected count information; and storing the corrected count information in the data structure.

[0137] Based on Example 21, use the method described in Example 1 to generate the embedding produced by Example 20.

[0138] Example 22 corresponds to any combination of Examples 1 through 21 above (e.g., any logically consistent permutation or subset).

[0139] Example 23 corresponds to any method counterpart, device counterpart, system counterpart, device plus function counterpart, computer-readable storage medium counterpart, data structure counterpart, article counterpart, graphical user interface presentation counterpart, etc., associated with Examples 1 through 22.

[0140] Finally, the functionality described herein can employ various mechanisms to ensure that any user data is handled in a manner that complies with applicable laws, social norms, and the expectations and preferences of individual users. For example, the functionality could allow users to explicitly opt in (and then explicitly opt out) of the functionality's provisions. The functionality could also provide appropriate security mechanisms to ensure the privacy of user data (e.g., data security mechanisms, encryption mechanisms, password protection mechanisms, etc.).

[0141] Furthermore, this specification may set forth various concepts in the context of an illustrative challenge or problem. This interpretation is not intended to imply that others should understand and / or articulate the challenge or problem in the manner specified herein. Moreover, this interpretation is not intended to imply that the subject matter of the claims is limited to solving the identified challenge or problem; that is, the subject matter of the claims may be applied in the context of challenges or problems other than those described herein.

[0142] Although the subject matter has been described using language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms of implementing the claims.

Claims

1. A computer-implemented method for generating a trained set of embeddings, comprising: Provide a set of training samples, each training sample describing a query submitted by at least one user, an item, and an indication of whether the item has been selected by the at least one user in response to submitting the query; Provide initial lexical embeddings associated with different training samples to multiple corresponding embedding update computing devices; Performing iterations of the training loop includes: The plurality of embedding update computing devices are used to generate a plurality of local lexical embedding sets; Provide the multiple local lexical embedding sets to multiple embedding merging computing devices; The plurality of embedding merging computing devices are used to generate a plurality of merged lexical embeddings from the plurality of local lexical embedding sets. A specific merged lexical embedding for a particular lexical represents the merging of two or more different local lexical embeddings computed by two or more corresponding embedding update computing devices for the same particular lexical; and The plurality of merged lexical embeddings are provided to the selected embedding update computing device for use by the selected embedding update computing device in the generation of local lexical embeddings in the next iteration of the training loop. The training loop is repeated until the training objective is achieved. When the training objective is achieved, the plurality of merged lexical embeddings provided to the embedding update computing device correspond to the trained embedding set. The trained set of embeddings is used in the inference phase, and the first prediction function used in the training loop to generate the plurality of local lexical embeddings is different from the second prediction function used in the inference phase and consumes more computational resources than the second prediction function. The first prediction function has at least global bias parameters and local bias parameters, and the second prediction function represents a distance metric.

2. The computer-implemented method of claim 1, wherein at least one of the plurality of embedded update computing devices is the same computing device as the embedded merging computing device.

3. The computer-implemented method of claim 1, wherein each particular training sample uses one or more query terms to represent a particular query, wherein each query term is represented as a query term vector having a dimension as large as the number of query terms in the vocabulary of the query terms.

4. The computer-implemented method of claim 1, wherein each particular training sample is represented by a term vector having a dimension as large as the number of terms in the set of selectable terms.

5. The computer-implemented method of claim 1, wherein each embedding update computing device generates a set of local lexical embeddings by means of the following steps: For specific training samples identifying a specific query and a specific item, generate a prediction indicating whether the specific query is considered similar to the specific item; and The local lexical embedding set is updated based on an indication of whether the prediction is consistent with a specific label associated with the specific training sample. Each embedded update computing device performs a process that depends on no more than one global parameter value, which is shared by processes performed by other embedded update computing devices.

6. The computer-implemented method of claim 1, wherein the specific merged lexical embedding for the specific lexical is generated by averaging together the two or more different local lexical embeddings for the same specific lexical.

7. The computer-implemented method of claim 1, wherein the provision of the plurality of merged lexical embeddings to the selected embedding update computing device involves sending a given merged lexical embedding associated with the given lexical to each embedding update computing device that has been assigned at least one training sample including the given lexical.

8. The computer-implemented method of claim 7, wherein an embedding update computing device that has not been assigned at least one training sample including a given lexical does not receive a given merged lexical embedding.

9. The computer-implemented method of claim 1, wherein the trained set of embeddings is used in the inference phase to determine a similarity score, the similarity score measuring the level of similarity between a query submitted by a user and candidate options.

10. The computer-implemented method of claim 1, wherein, in the training loop, different embedding update computing devices among the plurality of embedding update computing devices store different subsets of the complete set of lexical embeddings.

11. A computer-implemented method for determining a relationship between two or more items, comprising: Receive the identifier associated with the first and second items to be compared; Identify the first embedding associated with the first item; The identifier is the second embedding associated with the second item; The relationship between the first item and the second item is determined by determining the relationship between the first embedding and the second embedding in the vector space, so as to provide a similarity score; as well as Provides output results that depend on the similarity score. The first and second identifier embeddings utilize a data repository of trained embeddings generated by the training framework. The training framework uses a distributed computing device set and employs a first prediction function to generate the trained embeddings. This first prediction function differs from the second prediction function used in determining the relationship between the first and second terms, and consumes more computational resources than the second prediction function. The first prediction function includes at least global and local bias parameters, while the second prediction function represents a distance metric. The first item is a query submitted by the user via the user's computing device. The second option is one of the candidates under consideration. The identifier in the first embedding relates to a query embedding that identifies one or more terms associated with the query. The second embedded identifier relates to an item embedding that identifies the candidate. The determination of the relationship involves determining the distance between the query embedding and the item embedding, the distance corresponding to the similarity score, and The provision of the output results includes sending search results to the user computing device that identify the relationship between the candidate and the query.

12. The computer-implemented method of claim 11, wherein the identifier of the query embedding comprises: Multiple query components are embedded, each associated with a query term of the query. as well as The multiple query component embeddings are summed to generate the query embedding.

13. The computer-implemented method of claim 11, wherein the identifier of the item embedding comprises retrieving a single item embedding from a data repository associated with an item identifier that identifies the candidate.

14. The computer-implemented method of claim 11, wherein the identifier embedded in the item comprises: Embedded items that identify multiple terms that are associated with at least a portion of the candidate terms; as well as The multiple item component embeddings are summed to produce the item embedding.

15. The computer-implemented method of claim 14, wherein the item embedding is generated by summing the plurality of item component embeddings once it is determined that no single item embedding exists in the data repository associated with the item identifier that identifies the candidate.

16. The computer-implemented method of claim 14, wherein the plurality of item components are embedded in a heading associated with the candidate items to describe corresponding terms.

17. The computer-implemented method of claim 14, wherein different computing devices in the training framework store different subsets of the complete set of lexical embeddings.

18. A computer-implemented method, comprising: Access data structures; Receive raw count information for a specific target item from the data structure, the raw count information indicating that the specific target item has been selected for the first time in response to the query submission; Receive additional instances of raw count information about multiple other items from the data structure, each instance of the other raw count information indicating that another item has been selected multiple times in response to the same query submission; Identify the first embedding associated with the specific target item; Other embeddings associated with the other items are identified, and the first embedding and the other embeddings are generated using a first prediction function; The relationship between the first embedding and each of the other embeddings is determined in the vector space by using a second prediction function that is different from the first prediction function to determine the relationship between the first embedding and each of the other embeddings, so as to provide multiple similarity scores. Compared with the second prediction function, the first prediction function consumes more computational resources. The original count information for the specific target item is adjusted by adding other instances of the original count information as a weighted sum of the corresponding similarity scores of the plurality of similarity scores, to provide corrected count information. The corrected counting information is stored in the data structure; as well as The ranking of at least one of the other items in the search results provided to the user is adjusted relative to the specific target item based on the revised counting information.

19. A computing system having hardware logic circuitry configured to implement the computer-implemented method according to claim 1.

20. A computer-readable storage medium for storing computer-readable instructions, which one or more hardware processors execute to perform the computer-implemented method according to claim 1.

21. A computing system having hardware logic circuitry configured to implement the computer-implemented method according to claim 11.

22. A computer-readable storage medium for storing computer-readable instructions, which one or more hardware processors execute to perform the computer-implemented method of claim 11.

23. A computing system having hardware logic circuitry configured to implement the computer-implemented method according to claim 18.

24. A computer-readable storage medium for storing computer-readable instructions, which one or more hardware processors execute to perform the computer-implemented method of claim 18.

Images