Systems and methods for ranked document retrieval

Abstract

A computer implemented method for ranked document retrieval from a user query using weighted loss functions and categorical search queries. A system, device, or method implementing the computer implemented method may return documents that are ordered by the categories of the categorical search query such that there may be no post-processing document ranking stage.

Description

SYSTEMS AND METHODS FOR RANKED DOCUMENT RETRIEVALCROSS REFERENCE

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 562,642, filed March 7, 2024, the entirety of which is incorporated herein by reference.SUMMARY

[0002] Provided herein are methods, computer-readable media, and systems for efficient ranked document retrieval.

[0003] In one aspect, the embodiments herein disclose a method for ranking documents, comprising: receiving a query indicating user input; determining a query vector based on the query; retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents; determining, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and the at least one weight is determined based on one or more categories associated with the document of the set; ranking the set of the plurality of documents based on the score for each document of set; and providing the set of documents based on the ranking. In some embodiments, wherein the at least one weight is determined using a loss function. In some embodiments, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record. In some embodiments, wherein the at least one weight is determined based on a metric associated with the ranking. In some embodiments, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history. In some embodiments, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document. In some embodiments, wherein the machine learning model comprises a transformer model. In some embodiments, wherein the machine learning model is trained using a weighted cross entropy loss function. In some embodiments, wherein each category of the one or more categories is associated with a category value. In some embodiments, wherein a category of the one or more categories is prioritized when determining the at least one weight. In some embodiments, wherein prioritizing the category of the one or more categories is based on the metric. In someembodiments, further comprising providing the score for one or more of the set of documents. In some embodiments, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.

[0004] Another aspect of the present disclosure provides a non-transitory computer readable medium comprising machine executable code that, upon execution by one or more computer processors, implements any of the methods above or elsewhere herein.

[0005] Another aspect of the present disclosure provides a system comprising one or more computer processors and computer memory coupled thereto. The computer memory comprises machine executable code that, upon execution by the one or more computer processors, implements any of the methods above or elsewhere herein.INCORPORATION BY REFERENCE

[0006] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents and patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The novel features of the inventive concepts are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present inventive concepts will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the inventive concepts are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein), of which:

[0008] FIG. 1 shows a non-limiting example of a system for efficiently retrieving one or more document.

[0009] FIG. 2 shows a non-limiting example of a process for efficiently retrieving one or more ranked documents.

[0010] FIG. 3 shows a non-limiting example of pseudocode implementing a weighted cross entropy loss function.

[0011] FIG. 4 shows a non-limiting example of pseudocode implementing a multi-field weighted cross entropy loss function.

[0012] FIG. 5 shows a non-limiting example of imparting a weighting on a cosine similarity measure for a batch of left-hand side and right-hand side inputs.

[0013] FIG. 6 shows a non-limiting example of a computing device with one or more processors, memory, storage, and a network interface.

[0014] FIG. 7 shows a non-limiting example of a web / mobile application provision system providing browser-based and / or native mobile user interfaces.

[0015] FIG. 8 shows a non-limiting example of a cloud-based web / mobile application provision system comprising an elastically load balanced, auto-scaling web server and application server resources as well as synchronously replicated databases.

[0016] FIG. 9A shows two non-limiting examples of precision versus numbers of returned documents for various searching algorithms.

[0017] FIG. 9B shows two non-limiting examples of normalized discounted cumulative gain scores versus numbers of returned documents for various searching algorithms.

[0018] FIG. 10 shows an example search query and ranked returned documents.

[0019] FIG. 11 shows a set of example search queries and ranked returned documents.DETAILED DESCRIPTION

[0020] The systems, media, and methods disclosed herein are used to address long-standing limitations of document retrieval algorithms. The various embodiments disclosed herein provide at least improved, particular methods, systems, and media for efficient, relevant, ranked document retrieval. Some non-exhaustive benefits of the integration of the systems, media, and methods disclosed herein include increased document retrieval precision, decreased computational overhead, horizontal scalability, flexibility, live trend adaptiveness, and one-stage retrieval and ordering. The systems, media, and methods disclosed herein may improve searching algorithms (e.g., e-commerce or streaming service search functionalities) by imparting one-stage document retrieval and ordering, addressing the long felt need to reduce the computational overhead of disparate retrieval and ordering steps that may comprise extant keyword and vector-based searching algorithms.

[0021] As the size and complexity of databases grows ever larger, conventional methods of database searching and document retrieval are becoming increasingly resource intensive. Using conventional methods, the result of a database search, a set of documents (for example), may be returned as a result of matching one metric while requiring subsequent ordering or ranking based on one or more other metrics. The two-stage process of document retrieval and ranking increases computational overhead and challenges the scalability and adaptability of conventional searching methods. The systems, media, and methods disclosed herein enable efficient, ranked document retrieval (e.g., images, text, objects, etc ). A, system, media, or method implementing the techniques taught herein allow a client to pass a standard search query (e g., a type of product desired) and receive relevant documents according to a desired ranking that was determined based on aspects surrounding the user and / or the query. Accordingly, by efficiently encoding category-modified search queries and documents, the systems, media, and methods disclosed herein operate more efficiently to return ranked documents based on user search queries, requiring fewer resources than conventional methods.

[0022] In some embodiments, systems, media, and methods disclosed herein include a technique for retrieving ranked documents, comprising: receiving a query indicating user input; determining a query vector based on the query; retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents; determining, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and the at least one weight is determined based on one or more categories associated with the document of the set; ranking the set of the plurality of documents based on the score for each document of set; and providing the set of documents based on the ranking.Overview

[0023] In some embodiments of systems implementing the techniques disclosed herein, a system performing document retrieval may comprise the addition of weights to training functions of implemented machine learning algorithms so as to provide inherent ordering to the latent space of learned embeddings associated with documents and user queries. The latent space may be organized, and therefore inferable, by weights in loss functions used during training. The weights may comprise representations of categories comprising quality, relevance,personalization, reputation, recency, popularity, or query item record. The weights may further comprise another business metric from which a desired ranking may be established. The latent space comprising documents may be inferenced to obtain the documents most pertinent to the similarity of a query as measured by a similarity metric (e.g., cosine similarity). The similarity between a query and a document may be dependent on the weights applied to loss functions during training. Returned documents may then be inherently ordered by their relative similarity score. The document retrieval systems, methods, and media described herein may be trained using weighted loss functions in order to target a metric, where the metric may comprise business target performance metrics, which may include one or more of add to carts, revenue, user engagement, or conversion.

[0024] FIG. 1 shows a non-limiting example of a computing system 100 for requesting and retrieval of ranked documents. In this depicted embodiment, the system 100 comprises computing device 105 and processing device 125. In this depicted embodiment, the computing device 105 comprises the user interface 110, which comprises an input field 115. The input field 115 may be used for the input of a query 120. The user interface 110 may not comprise the input field 115. The computing device 105 and / or the user interface 110 may comprise a different element to receive query 120. The computing device may receive the query 120 from a local (e.g., another element of computing system 100) or non-local source (e.g., cloud server, remote server). In some embodiments, the computing device 105 serves as a client for a webapplication. The query 120 may be sent to a processing device 125, where the processing device 125 may be a cloud or remote server comprising a set of relevant documents (e.g., a video streaming server, a digital storefront server). In the depicted system of FIG. 1, the processing device 125 may comprise one or both of a document retrieval component 130 or a document store 135. The document retrieval component 130 may be a set of machine-readable instructions configured to receive the query 120 and execute the retrieval of documents from the document store 135. In some embodiments, the document retrieval component 130 may comprise one or more algorithms discussed elsewhere herein, where the one or more algorithms may be used for one or more of training or inferencing an algorithm to return documents in a ranked order. In some embodiments, the document store may be a database comprising at least the ranked documents 140 to be return based on the query 120. The document store 130 may be contained within the processing device 125. The document store may comprise one or more local or remote database, or a combination of local and remote databases. In some embodiments, the document retrieval component 130 may use the query 120 to obtain the plurality of documents from a document store 135. In some embodiments, the plurality of documents are associatedwith a plurality of vectors. The plurality of vectors may be compared to the query vector using a measure of similarity, where the measure of similarity may be a cosine similarity, and where the cosine similarity may be dependent on the one or more weights. Ultimately, the processing device 125 may return ranked documents 140 to the computing device 105.Inference

[0025] FIG. 2 shows a non-limiting example of an inference process 200 for the retrieval of ranked documents. In some embodiments, a computing device (e.g., computer device 105) may be configured to receive a query indicating user input at operation 210; determine a query vector based on the query at operation 220; retrieve a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents at operation 230; determine using a machine learning model a score for each document of a set of the plurality of documents at operation 240; rank the set of the plurality of documents based on the score for each document of the set at operation 250; and provide the set of documents based on the ranking at operation 260.

[0026] In some embodiments, the operation 210 may comprise receiving the query 120. In some embodiments, the query 120 may be input into the user interface 110 of the computing device 105. In some embodiments, the query 120 may comprise text, one or more images, one or more weights, one or more audio files, one or more videos, or any combination thereof. In some embodiments, the query 120 may comprise text. In some embodiments, the query 120 may comprise one or more images. In some embodiments, the query 120 may comprise one or more audio files. In some embodiments, the query 120 may comprise one or more videos. In some embodiments, the query may comprise text and one or more images. In some embodiments, the text may comprise an input of a user into the input field 115, where the text component of the query 120 indicates a written description, representation, or indication of the documents to be returned. In some embodiments, the one or more images may comprise an image-based input of a user into the input field 115, where the image component of the query 120 indicates a visual description, representation, or indication of the documents to be returned. In some embodiments, the one or more weights may comprise an indication of a document retrieval category (e.g., relevance, popularity, recency, quality, etc.), where the weight component of the query 120 indicates a categorical or scalar description, representation, or indication of the preference (e.g., score, rank) of documents returned.

[0027] In some embodiments, the operation 220 may comprise determining a query vector based on one or more components of the query 120. In some embodiments, the query vector isgenerated by one or more encoders. A query vector may be comprised of one or more components derived from text, image, audio, or video. Each component of a query may be passed through an appropriate encoder. For instance, a query may comprise of a text component, an image component, a video component, and an audio component, each of which may pass through an encoder trained for each type of component. In some embodiments, each vector output by an encoder may be combined into an overall query vector In some embodiments, text is converted to a text vector by a text encoder. In some embodiments an image is converted to an image vector by an image encoder. In some embodiments, the weights are converted to a weight vector by a weight encoder. In some embodiments, an audio file is converted to an audio vector by an audio encoder. In some embodiments, a video file is converted to a video vector by a video converter. In some embodiments, one or more of the text, image, audio, video, or weight encoder is a neural network trained to learn embeddings of the input types (e g., image, text, weight, audio, video). In some embodiments, the neural network converts each of the input types into vectors (e.g., numerical representations of the inputs) such that they may be compared directly for similarity. In some embodiments, one or more of the text vector, the image vector, the audio vector, the video, or the weight vector may be normalized (e.g., scaled to a uniform magnitude range) or otherwise processed. In some embodiments, one or more of the text vector, the image vector, or the weight vector are combined into a query vector. One skilled in the art will understand that encoders may be trained and vectors generated for other input types relevant to a search.

[0028] In some embodiments, the text vector may comprise a numerical representation of one or more instances of text. In some embodiments, the one or more instances of text may comprise one or more characters, a string, a phrase, one or more sentence, one or more paragraph, one or more clause, or other segment of a text input. In some embodiments, the image vector may comprise a numerical representation of the one or more images. In some embodiments, the weight vector may comprise vectors of one or more weights. In some embodiments, an audio file vector may comprise a numerical representation of one or more audio files. In some embodiments, a video file vector may comprise a numerical representation of one or more video files.

[0029] In some embodiments, the query vector may comprise a representation of the query that may be attributable to some region in a learned latent space. In some embodiments, the query vector may be an embedding of the query, where the vector may be associated with a latent space comprising a continuous representation of vectors. In some embodiments, the query vector or vector may comprise a similar representation as a plurality of document vectors or documentvectors. In some embodiments, the similarity in representation may be used to retrieve a plurality of documents.

[0030] In some embodiments, the operation 230 may comprise retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents. In some embodiments, the query 120 may be processed by a processing device 125, which may comprise a document retrieval component 130. In some embodiments, the document retrieval component 130 may use the query 120 to obtain the plurality of documents from a document store 135. In some embodiments, the plurality of documents are associated with a plurality of vectors. In some embodiments, the plurality of vectors may be compared to the query vector using a measure of similarity. In some embodiments, the measure of similarity may be a cosine similarity, where the cosine similarity may be dependent on the one or more weights.

[0031] In some embodiments, the operation 240 may comprise determining using a machine learning model a score for each document of a set of the plurality of documents. In some embodiments, a weighted cosine similarity between the query 120 and each of the set of the plurality of documents may be used to infer a score of each of the document vectors (and implicitly a ranking score for the documents), where the score of the documents may be associated with the weight categories including quality, relevance, personalization, reputation, recency, popularity, or query item record. In some embodiments, the score may be the basis of a ranking of the documents. For example, three documents may be returned with cosine similarities of 0.2, 7.9, and 5.2. In some embodiments, the documents would be returned in a ranked order of 7.9, 5.2, and 0.2. In some embodiments, the ranked order accounts for one or more categories. In some embodiments, the one or more categories may comprise a popularity or a seller reputation associated with the documents. Alternatively, without the techniques disclosed herein, the documents may be returned based on a similarity of query and document vector, where there is no consideration of the one or more categories except for in a secondary ranking operation. In some cases, the secondary ranking operation can be computationally expensive. In some embodiments, the latent space from which the plurality of documents were retrieved may have been learned by a machine learning model. In some embodiments, the latent space may have been learned based on one or more business metrics so as to retrieve the plurality of documents based on the metric, wherein the metric may include one or more of revenue, conversion (e g., a user action versus business goal, such as add to carts resulting in a purchase), user engagement, or item history. In some embodiments, the machine learning model may comprise an implementation of the similarity measure so as to determine a score for eachdocument of the set. In some embodiments, a score may comprise a measure of the angle (e.g., the cosine similarity) between the query vector and a document vector retrieved from the document store 135 In some embodiments, a threshold score may be applied to the similarity between the vector and each of the plurality of document vectors or each of the set of the plurality of document vectors such that the documents retrieved from the document store 135 are at least as similar to the query vector 120 as the threshold.

[0032] A similarity measured between a query and a document (or another type of pair) may be calculated using cosine similarity. Additionally, or alternatively, the similarity may be measured using another distance or angle measure. In some embodiments, the distance or angle measure may comprise a Euclidean distance, a Manhattan distance, a Jaccard similarity, a Hamming distance, a Mahalanobis distance, or other measure of the similarity between two vectors.

[0033] In some embodiments, the one or more categories comprise quality, relevance, personalization, reputation, recency, popularity, or query item record. In some embodiments, quality may include a measure based on user feedback (e.g., star ratings). In some embodiments, personalization may include one or more metric manually selected by a user or one or more metric automatically applied for a user (e.g., based on user preference or searching history). In some embodiments, reputation may include a measure of seller reliability (e.g., functioning product, responsiveness). In some embodiments, recency may include a measure of how long a document has been stored in one or more database being searched. In some embodiments, popularity may include a measure how often a document is interacted with (e.g., viewed, added to cart, purchased). In some embodiments, query time record may include one or more metric describing how a document has been interacted with in the past by one or more users. In some embodiments, relevance may comprise one or more of the other categories and be dependent on a custom combination of the one or more categories. In some embodiments, the custom combination comprises an equal representation of each category of the combination. In some embodiments, the categories of the combination are not equally represented (e.g., popularity is considered more than recency).

[0034] In some embodiments, the operation 250 may comprise ranking the set of the plurality of documents based on the score for each document of the set. In some embodiments, the ranking may be based on the score as calculated from the similarity measure. In some embodiments, the similarities of the set of the plurality of documents serves as the basis of the ranking such that no subsequent ranking of the set is necessary. In some embodiments, a higher score may be morerelevant. In some embodiments, a lower score may be more relevant. In some embodiments, a custom definition of relevancy may be applied to order the returned documents.

[0035] In some embodiments, the operation 260 may comprise providing the set of documents based on the ranking. In some embodiments, the set of documents comprise the ranked documents 140, and the ranked documents 140 are returned from the processing device 125 to the computing device 105. In some embodiments, the ranked documents are displayed on the user interface 110 of the computing device 105 in ranked order without the need to order or rank the documents on the computing device 105.

[0036] In some embodiments, the query 120 and / or the ranked documents 140 may comprise multiple components. The query 120 may comprise one or more image, one or more texts, one or more weights, one or more audio files, one or more videos, or any combination thereof. The ranked documents may comprise one or more image, one or more text, one or more weight, one or more audio file, one or more video, or any combination thereof. Using a query with multiple components (e.g., multi-modal) to obtain ranked documents that also contain multiple components (e.g., multi-modal) may provide particular utility over other techniques. Keyword based searching algorithms are not readily amenable to multi-modal, complex search queries (e.g., a query containing text, image, and audio components). Other machine-learning based searching algorithms may not provide the ability to return ranked documents. Further, techniques that rely on a secondary ranking operation may have computational limitations that scale with the complexity of returned documents (e.g., a document containing text, image, and audio components).Training

[0037] In some embodiments, a machine learning model may be trained to provide ranked documents based on a query (e.g., query 120). In some embodiments, the machine learning model may be disposed on the computing device 105. In some embodiments, the machine learning model may be disposed on the processing device 125. In some embodiments, the machine learning model may be disposed on the documented retrieval component 130. In some embodiments, the machine learning model may be trained on a weighted loss function such that the machine learning model encodes the underlying semantic information of a training dataset. In some embodiments, the semantic information comprises a mechanism for discerning preferred ranking among the documents. In some embodiments, the mechanism for discerning preferred ranking may be a similarity (or distance) measure, wherein the similarity measure can be used to order documents based on the similarity of the query with the documents. In someembodiments, a document with the highest vector (or embedding) similarity to the query is returned first in an order list, object, or data structure.

[0038] In some embodiments, the ranked documents may be returned by the machine learning model. In some embodiments, the input into the machine learning model may be the query and the output may be the ranked documents. In some embodiments, the machine learning model implements a contrastive learning approach. In some embodiments, contrastive learning approach applies supervised learning to train the machine learning model to associate one or more input types. In some embodiments, unsupervised, semi-supervised, or self-supervised learning may be used to train the machine learning model. In some embodiments, contrastive learning applies self-supervised learning to learn associations among inputs (e.g., documents and queries) without external labels. In some embodiments, an unsupervised learning approach may be used. In some embodiments, a semi-supervised learning approach may be used. In some embodiments, the one or more input types comprise queries or documents, where the queries or documents may be provided in pairs such that a query corresponds to at least a portion of a document. In some embodiments, the contrastive learning approach may seek to determine the cosine similarity between a query vector of the query and a document vector of the document. In some embodiments, the query vector may be compared with other documents, and the contrastive learning approach may seek to maximize the predicted cosine similarity of the query vector and the document vector while minimizing the similarity of the query vector and each of the other documents. In some embodiments, each portion of text may be associated with an image of the images, with the goal of the contrastive learning approach to organize latent space such that the portions of text and their associated images comprise a similar region of the latent space. In some embodiments, the contrastive learning approach may convert the cosine similarity between a query vector and a document vector to scaled logits for input into a softmax classifier such that a cross entropy loss, loss function may be applied to the training of the machine learning algorithm implementing the contrastive learning approach. In some embodiments, the similarity between a query vector and a document vector may be transformed by weights provided to a weighted cross entropy loss function so as to modulate the loss function by some weight category. In some embodiments, the weighted cross entropy loss function may be implemented to organize the latent space learned by the contrastive learning approach to account for one or more categories such as quality, relevance, personalization, reputation, recency, popularity, or query item record. In some embodiments, these concepts may comprise inputs of images, text, documents, objects, or weights.

[0039] In some embodiment, the contrastive learning approach may be implemented with a deep neural network (DNN). In some embodiments, the DNN may comprise convolutional layers. In some embodiments, the DNN may comprise a transformer architecture. In some embodiments, the DNN may comprise self-attention mechanisms. In some embodiments, the DNN may comprise one or more of a document encoder, an image encoder, a text encoder, a query encoder, or a weight encoder. In some embodiments, the DNN may take as input the query 120 and output the ranked documents 140.

[0040] In some embodiments, the DNN may implement a weighted loss function for training of the DNN. In some embodiments, the DNN may be embodied in the processing device 125 of the computing system 100 as shown in FIG. 1. FIG. 3 shows one non-limiting example of learning paradigm 300 implementing a weighted loss function. At operation 310, an input may be input into a machine learning model (e.g., DNN). This input may be obtained at the user interface 110 of the computing device 105. In some embodiments, the input may be passed to the document retrieval component 130 of the processing device 125, where the DNN may be implemented in the document retrieval component 130. In some embodiments, the input may comprise N triplets to be used for training the DNN. In some embodiments, the N triplets may comprise queries (e.g., images, text, audio, video), documents (e.g., images, text, audio, video), and weights. In some embodiments, there may be N quartets, N quintets, N sextets, N septets, N octets, N nonets, N decets, or greater groupings of fields. The multi-field nature of the queries and documents discussed herein may provide particular utility over other document retrieval techniques that do not allow for multi-field queries or documents (e.g., standard CLIP). In some embodiments, one or both of a query or a document may encompass multiple fields. For example, a query comprise one or more of a text component, an image component, a video component, or an audio component. Additionally, the text component may comprise one or more sub-components. In some embodiments, the one or more sub-components may be a title, a description, a date, a JSON metadata file, or other text-based String. The ability to impart multiple fields on a query may provide particular utility over document retrieval options that constrain inputs to one field. In some embodiments, the queries and the documents may be encoded by encoders in operation 320. In some embodiments, the encoded queries or encoded documents may by normalized at operation 330 to generate query vectors and document vectors. In some embodiments, the dot product may be calculated among the query vectors and the document vectors at operation 340. In some embodiments, labels may be computed at operation 350 for each of the dot-products, such that the query vectors and the appropriate document vectors receive a label, where the label identifies a unique query vector and document vectorpair. In some embodiments, the labels may comprise a one-hot encoded vector, wherein the vector comprises an array zeros of length N, and wherein the vector comprises a single one value in place of one zero value when the vector is used to identify a unique query vector and document vector pair, such that each index of the array corresponds to a unique query vector and document vector pair. In some embodiments, a loss may be calculated at operation 360, where a loss provides a metric for the difference between expected output and actual output, where the loss may be used to train the DNN to obtain a better organized latent space or a higher accuracy in prediction (e.g., between queries and documents). In some embodiments, the loss may be derived from a function In some embodiments, the function may comprise a cross-entropy loss function, a sigmoid loss function, a hinge loss function, or a log softmax function. One skilled in the art will recognize that various loss functions can be applied to measure an expected output versus a realized output and that the systems, media, and methods disclosed herein may be applied with such loss functions. In some embodiments, the loss function may be calculated for each query vector and document vector pair. In some embodiments, the loss may be a numerical measure corresponding to a numerical similarity between an output and some expected output. The loss may result in a minimizable value that may be used to train a machine learning model In some embodiments, the loss may between a set of documents and a set of queries may be minimized during training. In some embodiments, weights may be applied to said loss functions. The weights used in loss functions may be used to impart order in a latent space that may be queried using weights of a query. In some embodiments, the weights of a query may comprise a category and / or business metric (e.g., popularity, relevancy, item history, etc ). In some embodiments the similarity may be a function of the current training status of the DNN. In some embodiments, the similarity between two vectors may be a function of the vectors’ similarity in the latent space. In some embodiments, the loss may further comprise the weights, where the weights may be a modulating factor for each of the loss calculations for each of the query vector and document vector pairs. In some embodiments, the sum of each of the loss calculations for each of the query vector and document vector pairs may be used to compute the loss. In some embodiments, the negative value of the loss may be used to provide an appropriately minimizable function for back-propagation. In some embodiments, the loss may be used in operation 370 to update one or more of the DNN, the query encoder, or the document encoder. In some embodiments, the update may comprise back-propagation. In some embodiments, the loss may be used to update a machine learning model that implements a contrastive learning approach.

[0041] In some embodiments, the DNN may implement a weighted loss function for training of the DNN. FIG. 4 shows one non-limiting example of learning paradigm 400 implementing multiple fields and implementing a weighted loss function. In some embodiments, the multiple fields may comprise queries, documents, weights, images, text, or objects (vectorizable representations of data). At operation 405, an input (e.g., query 120) may be input into a machine learning model (e g., DNN) This input may be obtained at the user interface 110 of the computing device 105. In some embodiments, the input may be passed to the document retrieval component 130 of the processing device 125, where the DNN may be implemented in the document retrieval component 130. In some embodiments, the input may be used for training the DNN. In some embodiments, training may comprise a plurality of N queries (e.g., images, text), documents (e.g., objects), and weights. In some embodiments, the input may comprise a batch of N triplets, where the N triplets may comprise a plurality of left-hand fields and a plurality of right-hand fields (left and right used for clarity and differentiation, not to indicate directionality). In some embodiments, the shape of the input may comprise a greater number of fields. In some embodiments, there may be N quartets, N quintets, N sextets, N septets, N octets, N nonets, N decets, or greater groupings of fields. In some embodiments, the left-hand fields may comprise one or more queries. In some embodiments, the right-hand fields may comprise one or more documents. In some embodiments, a portion of the weights may correspond to the left-hand side and a portion of the weights may correspond to the right-hand side. In some embodiments, the sum of the weights for the left-hand side may sum to one. In some embodiments, the sum of the weights for the right-hand side may sum to one. In some embodiments, an each of the left-hand side fields may be associated with an encoder and each of the right-hand side fields may be associated with an encoder. In some embodiments, in operation 410, the left-hand side encoders may be used to extract vectors of the left-hand side fields. In some embodiments, in operation 415, the right-hand side encoders may be used to extract vectors of the and right-hand side fields. In some embodiments, the vectors for each of the lefthand side and each of the right-hand side fields may be normalized to obtain left-hand side vectors and right-hand side vectors in operation 420. In some embodiments, a data structure (e.g., a list, set, dictionary, etc.) may be initialized in operation 425. In some embodiments, the data structure may be populated with the pairwise dot products of each of the combinations of the left-hand side and the right-hand side vectors. In some embodiments, in operation 430, the left-hand side weights may be used to modulate one or both of the left-hand side vectors or vectors and the right-hand side weights may be used to modulate one or both of the right-hand side vectors or vectors. In some embodiments, the sum of the weighted left-hand side vectors or vectors may be transposed with the sum of the weighted right-hand side vectors or vectors. Insome embodiments, the transposition may comprise the mean of the left-hand side and the righthand side vectors or vectors. In some embodiments, the mean is added to the data structure. In some embodiments, labels may be computed at operation 435 for each of members of the data structure, such that each member of the data structure receives a label, where the label identifies a unique member. In some embodiments, the labels may comprise a one-hot encoded vector, where the vector comprises an array zeros of length N, and where the vector comprises a single one value in place of a single zero value when the vector is used to identify a unique member of the batch N, such that each index of the vector is associated with one member of the batch N. In some embodiments, at operation 440, a loss may be calculated for each of the members of the data structure using log softmax function. In some embodiments, the loss may be the sum of all of the losses among the left-hand side and right-hand side vectors or vectors. In some embodiments, the loss may be an average loss. In some embodiments, the loss may be used in operation 445 to update one or more of the DNN or each of the left-hand side and right-hand side encoders. In some embodiments, the update may comprise back-propagation. In some embodiments, the loss is used to update a machine learning model that implements a contrastive learning approach.

[0042] In some embodiments, the weights used as input into the learning paradigm 300 or the learning paradigm 400 may represent one or more categories comprising quality, relevance, personalization, reputation, recency, popularity, or query item record. In some embodiments the weights may be determined by non-differentiable or differentiable machine learning methods so as provide weights for loss functions for use with a DNN that may be trained with the goal of returning ranked documents, where the DNN may have been trained to target a business metric comprising revenue, conversion, user engagement, or item history. In some embodiments, the weights are applied to a loss function comprising cross entropy loss. In some embodiments, the weights are determined to represent a category of the one or more categories by the evaluation of a trained DNN on a task. In some embodiments, the task comprises the performance of the trained DNN in achieving one or business metric. In some embodiments, the business metric comprises revenue, conversion, user engagement, or item history. In some embodiments, a category or a combination of the one or more categories may be represented via weights, where the weights may comprise a component of a query (e.g., query 120) such that the query comprises an input into a DNN and the expected DNN output is one or more documents in a ranked order of relevance to the one or more business metric. In some embodiments, the weights that achieve the one or more business metric may be determined by evaluation of the performance of the trained DNN on a task after training. In some embodiments, the weights thatachieve the one or more business metric may be determined by evaluation of the performance of the DNN during training, where the relevance of the one or more documents correlates to a loss that may be leveraged during training (e.g., during back-propagation) to select for weights that return the one or more documents relevant to the one or more business metric.

[0043] In some embodiments, similarity between a first input (e.g., query, text, image, audio file, video, document, object) and a second input (e.g., query, text, image, document, object) may be measured by cosine similarity. In some embodiments, the cosine similarity may comprise the dot product between the first input and the second input divided by the lengths of the first input and the second input. In some embodiments, the cosine similarity produces an angle which is a metric that describes the similarity between the first input and the second input, where the maximum angle or similarity score is one, the minimum is negative one indicating dissimilarity, and the value of zero indicates orthogonality. In some embodiments, the cosine similarity may be adjusted via weights that are used to indicate one or more categories that adjusts the similarity of the first input and the second input, where the one or more categories comprise quality, relevance, personalization, reputation, recency, popularity, or query item record. In some embodiments, the weights may be used to alter the relative similarities of the first input and the second input based on the weights, such that the first input and the second input may have a different angle between one another in the latent space during inference. In some embodiments, the adjustment of relative similarity between the first and second input may be used as the basis for ranking documents, such that the one or more categories result in an inherent ranking of documents upon inference that requires no computationally intensive ranking operation after inference.

[0044] In some embodiments a weighted cross-entropy loss function may be used. The form of the weighted cross entropy loss function may be as shown below in function (1). In some embodiments, the weighted cross-entropy loss function may be used embodied in a contrastive learning paradigm. In the funciton below, loss (L) may be a function of log softmax (G) calculations. For an n-class, n-sample classification problem, the cross-entropy loss between the dot product matrix Z and an identity matrix can be computed. In function (1), the ithclass corresponding to the ithsample may be considered as the ground truth. The same cross-entopy loss may computed with the transpose of Z (ZT) to ensure symmetry of similarity (e.g., document to query, query to document). In some embodiments, this variation of cross-entropy loss may assign greater penalties to rows and columns with higher weights, biasing the gradients leveraged during training towards prioritizing the correction of the higher weighted values. Thismay result in pairs deemed more relevant based on their weights being adjusted preferentially during the training process.

[0045] In some embodiments, a weighted sigmoid loss function may be used. The sigmoid loss may operate directly on individual pairs without relying on a normalization process. This may result in loss calculated for each pair being independent of other pairs of a dataset used for training. In some embodiments, the ground truth matrix may comprise diagonal elements of 1 and -1 everywhere else. To incorporate ranking signals, the diagonal elements of the ground truth matrix may be multiplied with one or more weights and calculated using function (2). In function (2), t and b may be logit scale and bias. In some embodiments, the weight matrix W may be constructed such that its diagonal entries may be equal to the weights (w), and all off- diagonal values may be set to -1.

[0046] FIG. 5 shows a non-limiting embodiment of an encoding module 500 and cosine similarities altered by one or more weights. In some embodiments, the weights may be applied to another similarity measure as disclosed herein. In some embodiments, the similarity measure may comprise a distance or angle measure among one or more inputs. As depicted in FIG. 5, a batch of inputs may comprise a mapping between a left-hand side input 505 and a right-hand side input 510. The left-hand side input 505 and / or the right-hand side input 510 may comprise one or more components. For instance, the left-hand side input 505 shown in FIG. 5 comprises an image component and two text components comprising of a title and a description. In some embodiments, the left-hand side input 505 and / or the right-hand side input 510 may comprise one or more other components such as one or more additional text component, one or more additional image component, one or more additional video component, or one or more additional audio component. In some embodiments, the left-hand side input and / or the right-hand side input may comprise one or more weights. In some embodiments, the one or more weights may be supplied separately from the left-hand side input 505 and the right-hand side input 510. As shown in FIG. 5, a left-hand side input may comprise a query and the right-hand side input may comprise a document (e.g., a document of ranked documents 140). In some embodiments, a cosine similarity of 1 may indicate a complete match between a left-hand side input and a righthand side input. In some embodiments, a value of 1 may be assumed between instances of a training batch. In the instance of training, the left-hand side input and the right-hand side inputs of a sample may have been paired as representative of one another. In some embodiments, each of a plurality of left-hand side input 505 may be passed through one or more encoder. As shown in FIG. 5, the left-hand side input 505 comprises multiple components. Accordingly, the text components of the left-hand side input 505 may be converted to text embeddings for use in lefthand side vectors by a text encoder 515 and the image component of the left-hand side input 505 may be converted to an image embedding for use in left-hand side vectors by an image encoder 520. The right-hand side input 10 comprises text components passed into the text encoder to generate right-hand side embeddings for use in right-hand side vectors. In some embodiments, a cosine similarity may be calculated for each of the left-hand side vectors and the right-hand side vectors, where the vectors are compared pairwise as shown in matrix 525. In some embodiments, the cosine similarity of the left-hand side and the right-hand side vectors of the sample may be calculated to be one, where each sample of the batch has a similarity of one. In some embodiments, the cosine similarity may be modified by one or more weights of one or more categories. In some embodiments, the weights may be used during a training operation to render a latent space representing the left-hand side vectors and the right-hand side vectors inferable with weighted queries representing one or more category of interest. In some embodiments, the weights used during training or inference may comprise a business metric. The one or more weights may be used to impart a weighting to the determined similarities as shown in the weighted ground truth matrix 530. In some embodiments, the weights may be derived from a set of ranking scores 535. In some embodiments, the ranking scores 535 may be converted to weights via a score converter 540. In some embodiments, the score converter 540 may comprise a function (including an identity operation) to convert the rankings to weights that may be applied to the matrix 525. In some embodiments, the one or more weights may be used to ensure that pairs of a training sample are adjusted preferentially during the training process. In some embodiments, the preferential association between a subset of input pairs may be used to organize a latent space according to a business metric and / or one or more categories. In some embodiments, this organization may impart an inherent ranking onto the samples of the batch that may be leveraged during inference.

[0047] In some embodiments, a row of matrices of 525 or 530 may represent the similarity between a left-hand side input of a sample and each of the plurality of right-hand side inputs from the batch. In some embodiments, the off-diagonal elements of matrices 525 or 530 may indicate an incorrect pairing between the a member of the left-hand side input 505 and each ofthe members of a right-hand side input 510 that were not originally paired with the member of the left-hand side input 505 component. In some embodiments, a DNN and one or more encoders of a DNN may use the cosine similarity of the true pairs as a metric for training the DNN and the one or more encoders. In some embodiments, the true pairs correlate to the pairings of the left-hand side inputs and the right-hand side inputs of the samples of the batch prior to input of the batch into the encoding module 500. In some embodiments, a target for training the DNN comprises maximizing the diagonal elements of matrices 525 or 530 while minimizing the off-diagonal elements, which correspond to correct and incorrect pairings of lefthand side inputs and right-hand side inputs of samples of the batch, respectively.

[0048] In some embodiments, the cosine similarity (e.g., angle) of two inputs may be indicative of their ultimate similarity in the latent space learned by the DNN. In some embodiments, one input may be used as input into the DNN with the learned latent space. In some embodiments, the input (e.g., image, text, query, document, or object), which may be transformed into a vector or vectors via an encoder, may be used to extract vectors or vectors of other categories (e.g., image, text, query, document, or object). In some embodiments, the extracted vectors may be based on the angle between the input and the extracted vectors, such that the extracted vectors with the lowest angle may be considered to be the most similar to the input. In some embodiments, the angle may correspond to a similarity between a query and a document, which may enable the use of the DNN for the searching of documents by queries. In some embodiments, the angle may be dependent on weights provided with the query, such that the weights comprise one or more categories that may be used to extract documents from the latent space that may have a high similarity to a weight modulated query. In some embodiments, the one or more categories comprise quality, relevance, personalization, reputation, recency, popularity, or query item record.

[0049] In some embodiments, the DNN may be embodied in a search engine such that the search engine may be used to find the most relevant documents for a given query. In some embodiments, the search engine may require only a single step for the return of the most relevant documents due to the weights corresponding to the one or more categories. In some embodiments, this may provide a computationally more efficient searching techniques as there may be no need for a secondary ranking operation, which can be a costly step due to the complexities of comparing documents of arbitrary complexity. In some embodiments, the systems, methods, and media described above may forego the secondary ranking operation and provide documents in ranked or order or in a manner such that ranking is trivial. In some embodiments, a trivial ranking may comprise sorting a data structure (e.g., list, dictionary, map,set), wherein the data structure may comprise a score (e.g., the angles of documents) and an identifier of the documents (e.g., a key, ID, or other unique identifier). In some embodiments, the sorting algorithm may have an average time complexity of less than n2(e g , nlog(n)). In some embodiments, the lack of or trivial secondary ranking operation may enable greater computational efficiency of search engines or other software that employs the systems, methods, or media disclosed herein In some embodiments, the ranked (ordered) documents may be displayed, for example on user interface 110 of computing device 105. In some embodiments, ranked documents may be retrieved from the document store 135 by the document retrieval component 130 of the processing device 125, where the DNN may be embodied in or encoded onto the document retrieval component 125.

[0050] In some embodiments, a system (e.g., device, platform, website, software) implementing the training techniques disclosed herein may incorporate data from the system over time. In some embodiments, the system may build refinement data based on a deployment of the system. In some embodiments, the deployment may comprise engaging with users or other software to obtain the refinement data comprising search queries, returned documents, or user feedback. In some embodiments, the user feedback may comprise scalar or categorical data related to the quality of the returned documents. In some embodiments, the user feedback may be supplemented with manual labelling of returned document quality or relevance. In some embodiments, the refinement data may be used to fine-tune the system via additional training of a machine learning algorithm embodied in the system. In some embodiments, the machine learning model may be adaptive or improve over time due to iterations of the fine-tuning. In some embodiments, the fine-tuning may be used to capture live trends. In some embodiments, a live trend may comprise a category discussed elsewhere, where the category may comprise popularity. In some embodiments, the fine-tuning may enable the system to provide adaptive search capabilities such that the system improves or maintains quality over time without manual intervention. In some embodiments, the fine-tuning may be accomplished as part of a pipeline comprising automated machine learning. In some embodiments, the automated machine learning may comprise a third-party software.

[0051] Referring to FIG. 6, a block diagram is shown depicting an exemplary machine that includes a computer system 600 (e g., a processing or computing system) within which a set of instructions can execute for causing a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components in FIG. 6 are examples only and do not limit the scope of use or functionality ofany hardware, software, embedded logic component, or a combination of two or more such components implementing particular embodiments.

[0052] Computer system 600 may include one or more processors 601, a memory 603, and a storage 608 that communicate with each other, and with other components, via a bus 640. The bus 640 may also link a display 632, one or more input devices 633 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc ), one or more output devices 634, one or more storage devices 635, and various tangible storage media 636. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 640. For instance, the various tangible storage media 636 can interface with the bus 640 via storage medium interface 626. Computer system 600 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0053] Computer system 600 includes one or more processor(s) 601 (e.g., central processing units (CPUs), general purpose graphics processing units (GPGPUs), or quantum processing units (QPUs)) that carry out functions. Processor(s) 601 optionally contains a cache memory unit 602 for temporary local storage of instructions, data, or computer addresses. Processor(s) 601 are configured to assist in execution of computer readable instructions. Computer system 600 may provide functionality for the components depicted in Fig. 6 as a result of the processor(s) 601 executing non-transitory, processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 603, storage 608, storage devices 635, and / or storage medium 636. The computer-readable media may store software that implements particular embodiments, and processor(s) 601 may execute the software. Memory 603 may read the software from one or more other computer-readable media (such as mass storage device(s) 635, 636) or from one or more other sources through a suitable interface, such as network interface 620. The software may cause processor(s) 601 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 603 and modifying the data structures as directed by the software.

[0054] The memory 603 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM 604) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phasechange random access memory (PRAM), etc.), a read-only memory component (e.g., ROM605), and any combinations thereof. ROM 605 may act to communicate data and instructions unidirectionally to processor(s) 601, and RAM 604 may act to communicate data and instructions bidirectionally with processor(s) 601. ROM 605 and RAM 604 may include any suitable tangible computer-readable media described below. In one example, a basic input / output system 606 (BIOS), including basic routines that help to transfer information between elements within computer system 600, such as during start-up, may be stored in the memory 603.

[0055] Fixed storage 608 is connected bidirectionally to processor(s) 601, optionally through storage control unit 607. Fixed storage 608 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 608 may be used to store operating system 609, executable(s) 610, data 611, applications 612 (application programs), and the like. Storage 608 can also include an optical disk drive, a solid-state memory device (e.g., flash-based systems), or a combination of any of the above. Information in storage 608 may, in appropriate cases, be incorporated as virtual memory in memory 603.

[0056] In one example, storage device(s) 635 may be removably interfaced with computer system 600 (e.g., via an external port connector (not shown)) via a storage device interface 625. Particularly, storage device(s) 635 and an associated machine-readable medium may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 600. In one example, software may reside, completely or partially, within a machine-readable medium on storage device(s) 635. In another example, software may reside, completely or partially, within processor(s) 601.

[0057] Bus 640 connects a wide variety of subsystems. Herein, reference to a bus may encompass one or more digital signal lines serving a common function, where appropriate. Bus 640 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures. As an example and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro Channel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCLX) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, serial advanced technology attachment (SATA) bus, and any combinations thereof.

[0058] Computer system 600 may also include an input device 633. In one example, a user of computer system 600 may enter commands and / or other information into computer system 600via input device(s) 633. Examples of an input device(s) 633 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof. In some embodiments, the input device is a Kinect, Leap Motion, or the like. Input device(s) 633 may be interfaced to bus 640 via any of a variety of input interfaces 623 (e.g., input interface 623) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.

[0059] In particular embodiments, when computer system 600 is connected to network 630, computer system 600 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 630. Communications to and from computer system 600 may be sent through network interface 620. For example, network interface 620 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 630, and computer system 600 may store the incoming communications in memory 603 for processing. Computer system 600 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 603 and communicated to network 630 from network interface 620. Processor(s) 601 may access these communication packets stored in memory 603 for processing.

[0060] Examples of the network interface 620 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of a network 630 or network segment 630 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof. A network, such as network 630, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used.

[0061] Information and data can be displayed through a display 632. Examples of a display 632 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) suchas a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display, a plasma display, and any combinations thereof. The display 632 can interface to the processor(s) 601, memory 603, and fixed storage 608, as well as other devices, such as input device(s) 633, via the bus 640. The display 632 is linked to the bus 640 via a video interface 622, and transport of data between the display 632 and the bus 640 can be controlled via the graphics control 621. In some embodiments, the display is a video projector. In some embodiments, the display is a headmounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting examples, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VR One, Avegant Glyph, Freefly VR headset, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0062] In addition to a display 632, computer system 600 may include one or more other peripheral output devices 634 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 640 via an output interface 624. Examples of an output interface 624 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

[0063] In addition or as an alternative, computer system 600 may provide functionality as a result of logic hardwired or otherwise embodied in a circuit, which may operate in place of or together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein. Reference to software in this disclosure may encompass logic, and reference to logic may encompass software. Moreover, reference to a computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware, software, or both.

[0064] Those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0065] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a generalpurpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration

[0066] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0067] In accordance with the description herein, suitable computing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Those of skill in the art will also recognize that select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations, known to those of skill in the art.

[0068] In some embodiments, the computing device includes an operating system configured to perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device’s hardware and provides services for execution of applications. Those of skill in the art will recognize that suitable server operating systems include, by way of non-limiting examples, FreeBSD, OpenBSD, NetBSD®, Linux, Apple® MacOS X Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in the art will recognize that suitable personal computer operating systems include, by way of nonlimiting examples, Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In some embodiments, the operating system is provided by cloud computing. Those of skill in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia® Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS, Linux®, and Palm® WebOS®. Those of skill in the art will also recognize that suitable media streaming device operating systems include, by way of non-limiting examples, Apple TV®, Roku®, Boxee®, Google TV®, Google Chromecast®, Amazon Fire®, and Samsung® HomeSync®. Those of skill in the art will also recognize that suitable video game console operating systems include, by way of non-limiting examples, Sony® PS3®, Sony® PS4®, Sony® PS5®, Microsoft® Xbox 360®, Microsoft® Xbox One, Microsoft® Xbox Series X, Microsoft® Xbox Series S, Nintendo® Wii®, Nintendo® Wii U®, Nintendo® Switch™, and Ouy ®.

[0069] Another aspect of the disclosure herein describes a non-transitory, computer-readable medium comprising executable instructions, wherein when a processor, when executing the executable instructions, performs a method as described herein.Web application

[0070] In some embodiments, a computer program includes a web application. In light of the disclosure provided herein, those of skill in the art will recognize that a web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some embodiments, a web application is created upon a software framework such as Microsoft® .NET or Ruby on Rails (RoR). In some embodiments, a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non-relational, object oriented, associative, XML, and document oriented database systems. In further embodiments, suitable relational database systems include, by way of non-limiting examples, Microsoft® SQL Server, mySQL™, and Oracle®. Those of skill in the art will also recognize that a web application, in various embodiments, is written in one or more versions of one or more languages. A web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML),Extensible Hypertext Markup Language (XHTML), or extensible Markup Language (XML). In some embodiments, a web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS) In some embodiments, a web application is written to some extent in a client-side scripting language such as Asynchronous JavaScript and XML (AJAX), Flash® ActionScript, JavaScript, or Silverlight®. In some embodiments, a web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python™, Ruby, Tel, Smalltalk, WebDNA®, or Groovy. In some embodiments, a web application is written to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, a web application integrates enterprise server products such as IBM® Lotus Domino®. In some embodiments, a web application includes a media player element. In various further embodiments, a media player element utilizes one or more of many suitable multimedia technologies including, by way of non-limiting examples, Adobe® Flash®, HTML 5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

[0071] Referring to FIG. 7, in a particular embodiment, an application provision system comprises one or more databases 700 accessed by a relational database management system (RDBMS) 710. Suitable RDBMSs include Firebird, MySQL, PostgreSQL, SQLite, Oracle Database, Microsoft SQL Server, IBM DB2, IBM Informix, SAP Sybase, Teradata, and the like. In this embodiment, the application provision system further comprises one or more application severs 720 (such as Java servers, .NET servers, PHP servers, and the like) and one or more web servers 730 (such as Apache, IIS, GWS and the like). The web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 740. Via a network, such as the Internet, the system provides browser-based and / or mobile native user interfaces.

[0072] Referring to FIG. 8, in a particular embodiment, an application provision system alternatively has a distributed, cloud-based architecture 800 and comprises elastically load balanced, auto-scaling web server resources 810 and application server resources 820 as well synchronously replicated databases 830.Mobile application

[0073] In some embodiments, a computer program includes a mobile application provided to a mobile computing device. In some embodiments, the mobile application is provided to a mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to a mobile computing device via the computer network described herein.

[0074] In view of the disclosure provided herein, a mobile application is created by techniques known to those of skill in the art using hardware, languages, and development environments known to the art. Those of skill in the art will recognize that mobile applications are written in several languages. Suitable programming languages include, by way of non-limiting examples, C, C++, C#, Objective-C, Java™, JavaScript, Pascal, Object Pascal, Python™, Ruby, Rails, VB.NET, WML, and XHTML / HTML with or without CSS, or combinations thereof

[0075] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non-limiting examples, AirplaySDK, alcheMo, Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework, Rhomobile, and WorkLight Mobile Platform. Other development environments are available without cost including, by way of non-limiting examples, Lazarus, MobiFlex, MoSync, and Phonegap. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting examples, iPhone and iPad (iOS) SDK, Android™ SDK, BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, and Windows® Mobile SDK.

[0076] Those of skill in the art will recognize that several commercial forums are available for distribution of mobile applications including, by way of non-limiting examples, Apple® App Store, Google® Play, Chrome WebStore, BlackBerry® App World, App Store for Palm devices, App Catalog for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo® DSi Shop.Standalone application

[0077] In some embodiments, a computer program includes a standalone application, which is a program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in. Those of skill in the art will recognize that standalone applications are often compiled. A compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C, COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB .NET, or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, a computer program includes one or more executable complied applications.Web browser plug-in

[0078] In some embodiments, the computer program includes a web browser plug-in (e g., extension, etc.). In computing, a plug-in is one or more software components that add specific functionality to a larger software application. Makers of software applications support plug-ins to enable third-party developers to create abilities which extend an application, to support easily adding new features, and to reduce the size of an application. When supported, plug-ins enable customizing the functionality of a software application. For example, plug-ins are commonly used in web browsers to play video, generate interactivity, scan for viruses, and display particular file types. Those of skill in the art will be familiar with several web browser plug-ins including, Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®. In some embodiments, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some embodiments, the toolbar comprises one or more explorer bars, tool bands, or desk bands.

[0079] In view of the disclosure provided herein, those of skill in the art will recognize that several plug-in frameworks are available that enable development of plug-ins in various programming languages, including, by way of non-limiting examples, C++, Delphi, Java™, PHP, Python™, and VB .NET, or combinations thereof.

[0080] Web browsers (also called Internet browsers) are software applications, designed for use with network-connected computing devices, for retrieving, presenting, and traversing information resources on the World Wide Web. Suitable web browsers include, by way of nonlimiting examples, Microsoft® Internet Explorer®, Mozilla® Firefox®, Google® Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. In some embodiments, the web browser is a mobile web browser. Mobile web browsers (also called microbrowsers, mini -browsers, and wireless browsers) are designed for use on mobile computing devices including, by way of nonlimiting examples, handheld computers, tablet computers, netbook computers, subnotebook computers, smartphones, music players, personal digital assistants (PDAs), and handheld video game systems. Suitable mobile web browsers include, by way of non-limiting examples, Google® Android® browser, RIM BlackBerry® Browser, Apple® Safari®, Palm® Blazer, Palm® WebOS® Browser, Mozilla® Firefox® for mobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web, Nokia® Browser, Opera Software® Opera® Mobile, and Sony® PSP™ browser.Software modules

[0081] In some embodiments, the systems, media, and methods disclosed herein include software, server, and / or database modules, or use of the same. In view of the disclosure provided herein, software modules are created by techniques known to those of skill in the art using machines, software, and languages known to the art The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a fde, a section of code, a programming object, a programming structure, a distributed computing resource, a cloud computing resource, or combinations thereof. In further various embodiments, a software module comprises a plurality of fdes, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, a plurality of distributed computing resources, a plurality of cloud computing resources, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, a standalone application, and a distributed or cloud computing application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on a distributed computing platform such as a cloud computing platform. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.Databases

[0082] In some embodiments, the systems, media, and methods disclosed herein include one or more databases, or use of the same. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of images, text, objects, documents, video, commercial products, or any combination thereof. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object-oriented databases, object databases, entity-relationship model databases, associative databases, XML databases, document-oriented databases, and graph databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, Sybase, and MongoDB. In some embodiments, a database is Internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloudcomputing based. In a particular embodiment, a database is a distributed database. In other embodiments, a database is based on one or more local computer storage devices.Data transmission

[0083] The subject matter described herein, including methods and systems as described herein and may be configured to be performed in one or more facilities at one or more locations. Facility locations are not limited by country and include any country or territory. In some instances, one or more steps are performed in a different country than another step of the method. In some embodiments, one or more method steps involving a computer system are performed in a different country than another step of the methods provided herein. In some embodiments, data processing and storage are performed in a different country or location than one or more steps of the methods described herein. In some embodiments, one or more products or data are transferred from one or more of the facilities to one or more different facilities for analysis or further analysis. Data includes, but is not limited to, information regarding the stratification of a subject, and any data produced by the methods disclosed herein. In some embodiments of the methods and systems described herein, the subject information is compiled, and a subsequent data transmission step will transmit or store the subject information.

[0084] In some embodiments, any step of any method described herein is performed by a software program or module on a computer. In additional or further embodiments, data from any step of any method described herein is transferred to and from facilities located within the same or different countries, including analysis performed in one facility in a particular location and the data shipped to another location or directly to an individual in the same or a different country. In additional or further embodiments, data from any step of any method described herein is transferred to and / or received from a facility located within the same or different countries, including analysis of a data input, such as queries, objects, properties, types, filters, tables, or any combination thereof, performed in one facility in a particular location and corresponding data transmitted to another location.Business methods utilizing a computer

[0085] The methods described herein may utilize one or more computers. The computer may be used for managing customer and subject information such as a documents database. The computer may include a monitor or other user interface for displaying data, results, billing information, marketing information (e.g., demographics), customer information, or sample information. The computer may also include means for data or information input. The computermay include a processing unit and fixed or removable media or a combination thereof. The computer may be accessed by a user in physical proximity to the computer, for example via a keyboard and / or mouse, or by a user that does not necessarily have access to the physical computer through a communication medium such as a modem, an internet connection, a telephone connection, or a wired or wireless communication signal carrier wave. In some cases, the computer may be connected to a server or other communication device for relaying information from a user to the computer or from the computer to a user. In some cases, the user may store data or information obtained from the computer through a communication medium on media, such as removable media. It is envisioned that data relating to the methods can be transmitted over such networks or connections for reception and / or review by a party.

[0086] The entity entering or reviewing information into a database for the purpose of one or more of the following: inventory tracking, order tracking, customer management, customer service, billing, and sales. Sample information may include, but is not limited to: customer name, unique customer identification, or any information suitable for storage in a database.

[0087] The database may be accessible by a user. Database access may take the form of electronic communication such as a computer or telephone. The database may be accessed through an intermediary such as a customer service representative, business representative, or consultant. The availability or degree of database access may change upon payment of a fee for products and services rendered or to be rendered.Definitions

[0088] Unless defined otherwise, all terms of art, notations and other technical and scientific terms or terminology used herein are intended to have the same meaning as is commonly understood by one of ordinary skill in the art to which the claimed subject matter pertains. In some cases, terms with commonly understood meanings are defined herein for clarity and / or for ready reference, and the inclusion of such definitions herein should not necessarily be construed to represent a substantial difference over what is generally understood in the art.Illustrative Embodiments

[0089] Among the exemplary embodiments are:

[0090] Embodiment 7: A computer-implemented method for ranking documents, comprising: a. receiving a query indicating user input; b. determining a query vector based on the query;c. retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents; d. determining, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: i. the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and ii. the at least one weight is determined based on one or more categories associated with the document of the set; e. ranking the set of the plurality of documents based on the score for each document of set; and f. providing the set of documents based on the ranking.

[0091] Embodiment 2'. The method of embodiment 1, wherein the at least one weight is determined using a loss function.

[0092] Embodiment 3: The method of embodiment 1 or 2, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record.

[0093] Embodiment 4'. The method of any one of embodiments 1 to 3, wherein the at least one weight is determined based on a metric associated with the ranking.

[0094] Embodiment 5: The method of any one of embodiments 1 to 4, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history.

[0095] Embodiment 6: The method of any one of embodiments 1 to 5, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document.

[0096] Embodiment 7: The method of any one of embodiments 1 to 6, wherein the machine learning model comprises a transformer model.

[0097] Embodiment 8: The method of any one of embodiments 1 to 7, wherein the machine learning model is trained using a weighted cross entropy loss function.

[0098] Embodiment 9: The method of any one of embodiments 1 to 8, wherein each category of the one or more categories is associated with a category value.

[0099] Embodiment 10 The method of any one of embodiments 1 to 9, wherein a category of the one or more categories is prioritized when determining the at least one weight.

[0100] Embodiment 11: The method of any one of embodiments 1 to 10, wherein prioritizing the category of the one or more categories is based on the metric.

[0101] Embodiment 12 : The method of any one of embodiments 1 to 11, further comprising providing the score for one or more of the set of documents.

[0102] Embodiment 13: The method of any one of embodiments 1 to 12, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.

[0103] Embodiment 14: A system comprising: a. at least one memory; and b. one or more processors; wherein the at least one memory comprises computer-readable instructions which, when executed, cause the at least one or more processors to cause the system to: i. receive a query indicating user input, ii. determine a query vector based on the query, iii. retrieve a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents, iv. determine, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: a. the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, andb. the at least one weight is determined based on one or more categories associated with the document of the set; v. rank the set of the plurality of documents based on the score for each document of set, and vi. provide the set of documents based on the ranking.

[0104] Embodiment 15: The system of embodiment 14, wherein the at least one weight is determined using a loss function.

[0105] Embodiment 16: The system of embodiment 14 or 15, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record.

[0106] Embodiment 17: The system of any one of embodiments 14 to 16, wherein the at least one weight is determined based on a metric associated with the ranking.

[0107] Embodiment 18: The system of any one of embodiments 14 to 17, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history.

[0108] Embodiment 19: The system of any one of embodiments 14 to 18, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document.

[0109] Embodiment 20: The system of any one of embodiments 14 to 19, wherein the machine learning model comprises a transformer model.

[0110] Embodiment 21: The system of any one of embodiments 14 to 20, wherein the machine learning model is trained using a weighted cross entropy loss function.

[0111] Embodiment 22: The system of any one of embodiments 14 to 21, wherein each category of the one or more categories is associated with a category value.

[0112] Embodiment 23: The system of any one of embodiments 14 to 22, wherein a category of the one or more categories is prioritized when determining the at least one weight.

[0113] Embodiment 24: The system of any one of embodiment 14 to 23, wherein prioritizing the category of the one or more categories is based on the metric.

[0114] Embodiment 25: The system of any one of embodiments 14 to 24, further comprising providing the score for one or more of the set of documents.

[0115] Embodiment 26: The system of any one of embodiments 14 to 25, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.

[0116] Embodiment 27'. A non-transitory, computer-readable medium comprising executable instructions, wherein when one or more processors, when executing the executable instructions, performs a method for ranking documents, the method comprising: a. receiving a query indicating user input; b. determining a query vector based on the query; c. retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents; d. determining, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: i. the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and ii. the at least one weight is determined based on one or more categories associated with the document of the set, e. ranking the set of the plurality of documents based on the score for each document of set; and f. providing the set of documents based on the ranking.

[0117] Embodiment 28: The computer-readable medium of embodiment 27, wherein the at least one weight is determined using a loss function.

[0118] Embodiment 29: The computer-readable medium of embodiment 27 or 28, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record.

[0119] Embodiment 30: The computer-readable medium of any one of 27 to 29, wherein the at least one weight is determined based on a metric associated with the ranking.

[0120] Embodiment 31: The computer-readable medium of any one of embodiments 27 to 30, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history.

[0121] Embodiment 32: The computer-readable medium of any one of embodiments 27 to 31, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document.

[0122] Embodiment 33: The computer-readable medium of any one of embodiments 27 to 32, wherein the machine learning model comprises a transformer model.

[0123] Embodiment 34: The computer-readable medium of any one of embodiments 27 to 33, wherein the machine learning model is trained using a weighted cross entropy loss function.

[0124] Embodiment 35: The computer-readable medium of any one of embodiments 27 to 34, wherein each category of the one or more categories is associated with a category value.

[0125] Embodiment 36: The computer-readable medium of any one of embodiments 27 to 35, wherein a category of the one or more categories is prioritized when determining the at least one weight.

[0126] Embodiment 37: The computer-readable medium of any one of embodiments 27 to 36, wherein prioritizing the category of the one or more categories is based on the metric.

[0127] Embodiment 38: The computer-readable medium of any one of embodiments 27 to 37, further comprising providing the score for one or more of the set of documents.

[0128] Embodiment 39: The computer-readable medium of any one of embodiments 27 to 38, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.EXAMPLES

[0129] The following examples are included for illustrative purposes only and are not intended to limit the scope of the inventive concepts.Example 1: Performance of Weighted Contrastive Learning

[0130] The weighting described herein was applied to CLIP (contrastive language-image pretraining), a contrastive learning paradigm initially intended to match images with text descriptions. CLIP was initially shown to employ and image encoder and a text encoder to learn vectors of images and text. The images and text were paired, such that CLIP was trained to maximize the cosine similarity of vectors of images and vectors of text that were originallypaired, while minimizing the cosine similarity of vectors of images and texts that were not originally paired. In this way, CLIP could then be leveraged to search a large corpus of images and text to find the most appropriate match for an image or text, whether or not the image or text used on the trained CLIP model was a member for the pairs used for training. This was useful, for example, in image captioning, where the cosine similarity of a vector of an image and a variety of text vectors could be compared such that the text vector with the highest cosine similarity to the image embedding would be returned as the most appropriate caption for the image.

[0131] The implementation of the system, media, and methods discussed elsewhere herein with algorithms such as CLIP, for example, have been shown to provide a significant improvement to performance. Evaluations of various algorithms both foregoing and implementing the system, media, and methods discussed elsewhere herein are shown in FIG. 9A and FIG. 9B. In FIG. 9A and FIG. 9B, four graphs are shown to exemplify the improved performance of algorithms implementing weights as discussed elsewhere herein.

[0132] FIG. 9A shows the performance of four machine learning models, where three implement the training approaches discussed elsewhere herein, and one (the bottom curve) shows the pre-trained CLIP model performance. The performance metric shown on the left 905 is the precision of the various models versus the number of items requested (k), where the performance of the models is measured versus training data (200k possible items). The precision, in this instance, may be calculated based on the ratio of true returned results versus the number of returned results. For example, for k = 10, a model is asked to return 10 results for a given query. If 7 of the 10 return results are true top 10 results, then the precision would be calculated to be 7. It can be seen that the top three lines, indicative of models implementing the training approach discussed elsewhere herein, outperform the pre-trained CLIP model for every value of k. The same trend can be seen in the right 910 panel, where the models were measured versus validation data (200k possible items).

[0133] FIG. 9B shows the performance of four machine learning models, where three implement the training approaches discussed elsewhere herein, and one (the bottom curve) shows the pre-trained CLIP model performance. The metric shown in the left 915 is the normalized discounted cumulative gain (NDCG) versus the number of items requested (k), where the performance of the models is measured versus training data (200k possible items). The NDCG, in this instance, may be calculated to reflect the ranking quality of returned results for searching algorithms. In an NDCG calculation, the relevance of the documents returned iscompared with an ideal ranking (e.g., by a perfect search). For each returned document, a relevance score of each document is discounted by its position in the returned search, and the relevance score of each document is summed and divided by the maximum possible discounted score (e.g., the score of the documents returned in the ideal order). In FIG. 9B, it can be seen that pre-trained CLIP NDCG does not change as k increases as the relevance of documents returned is not dependent on the weighting methods discussed herein, so the documents of regular CLIP will always be returned in a “unranked” manner such that the first document is not guaranteed to be more relevant than the last. Alternatively, it can be seen in the top three lines, which implement the training methods discussed elsewhere herein, that the NDCG shows improved NDCG values as k increases. This phenomenon demonstrates the one-step document retrieval and ranking performance imparted by the systems, media, and methods disclosed herein. The same trend can be seen in the right 920 panel, where the models were measured versus validation data (200k possible items).Example 2: Search query and returned ranked documents

[0134] A search is conducted using multiple categories as weights for a search query to return documents pertaining to “basketball” shirt, as depicted in FIG. 10. The search returns ten of the most relevant basketball shirts for a text-based query modified by weights corresponding to seller reputation and popularity. The returned documents, which in this example comprise an image of the shirt, the retailer’s name, a brief description of the shirt, the price of the shirt, sale information regarding the shirt, and a feedback mechanism (the heart) for a user’s perception of quality of the document relative to their search query. In this case, the most left t-shirt of the top row is found to be the most relevant t-shirt for the search query “basketball shirt” as modified by the categories of seller-reputation and popularity.Example 3: Ranked Search Results for Product Queries

[0135] A search was conducted for various fashion and home goods. The results are shown in FIG. 11. A model is trained using a weighted contrastive learning paradigm using a Google Shopping results order as a proxy for a business metric. Each of the queries shown return a plurality of documents with inherent scores due to said training. For example, the shoes in the top left of FIG. 11 are predicted to correlate most strongly with the proxy business metric and received a score of 100. Alternatively, the shoes in the bottom right portion the image associated with the query “Vintage Style Women’s Oxfords” are predicted as less relevant to the proxy business metric while still being relevant to the text-based portion of the query. In this way, highly similar results (in a visual sense) are ordered based on a non-obvious metric that can beused to align a business goal with a searching paradigm that leverages ranked document retrieval.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A computer-implemented method for ranking documents, comprising: a. receiving a query indicating user input; b. determining a query vector based on the query; c. retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents; d. determining, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: i. the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and ii. the at least one weight is determined based on one or more categories associated with the document of the set; e. ranking the set of the plurality of documents based on the score for each document of set; and f. providing the set of documents based on the ranking.

2. The method of claim 1, wherein the at least one weight is determined using a loss function.

3. The method of claim 1 or 2, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record.

4. The method of any one of claims 1 to 3, wherein the at least one weight is determined based on a metric associated with the ranking.

5. The method of any one of claims 1 to 4, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history6. The method of any one of claims 1 to 5, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document.

7. The method of any one of claims 1 to 6, wherein the machine learning model comprises a transformer model.

8. The method of any one of claims 1 to 7, wherein the machine learning model is trained using a weighted cross entropy loss function.

9. The method of any one of claims 1 to 8, wherein each category of the one or more categories is associated with a category value.

10. The method of any one of claims 1 to 9, wherein a category of the one or more categories is prioritized when determining the at least one weight.

11. The method of any one of claims 1 to 10, wherein prioritizing the category of the one or more categories is based on the metric.

12. The method of any one of claims 1 to 11, further comprising providing the score for one or more of the set of documents.

13. The method of any one of claims 1 to 12, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.

14. A system comprising: a. at least one memory; and b. one or more processors; wherein the at least one memory comprises computer-readable instructions which, when executed, cause the at least one or more processors to cause the system to: i. receive a query indicating user input, ii. determine a query vector based on the query, iii. retrieve a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectorsassociated with the plurality of documents corresponds to a document of the plurality of documents, iv. determine, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: a. the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and b. the at least one weight is determined based on one or more categories associated with the document of the set; v. rank the set of the plurality of documents based on the score for each document of set, and vi. provide the set of documents based on the ranking.

15. The system of claim 14, wherein the at least one weight is determined using a loss function.

16. The system of claim 14 or 15, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record.

17. The system of any one of claims 14 to 16, wherein the at least one weight is determined based on a metric associated with the ranking.

18. The system of any one of claims 14 to 17, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history.

19. The system of any one of claims 14 to 18, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document.

20. The system of any one of claims 14 to 19, wherein the machine learning model comprises a transformer model.

21. The system of any one of claims 14 to 20, wherein the machine learning model is trained using a weighted cross entropy loss function.

22. The system of any one of claims 14 to 21, wherein each category of the one or more categories is associated with a category value.

23. The system of any one of claims 14 to 22, wherein a category of the one or more categories is prioritized when determining the at least one weight.

24. The system of any one of claim 14 to 23, wherein prioritizing the category of the one or more categories is based on the metric.

25. The system of any one of claims 14 to 24, further comprising providing the score for one or more of the set of documents.

26. The system of any one of claims 14 to 25, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.

27. A non-transitory, computer-readable medium comprising executable instructions, wherein when one or more processors, when executing the executable instructions, performs a method for ranking documents, the method comprising: a. receiving a query indicating user input; b. determining a query vector based on the query; c. retrieving a plurality of documents based on angles between the query vector and a plurality of vectors associated with the plurality of documents, wherein each vector of the plurality of vectors associated with the plurality of documents corresponds to a document of the plurality of documents; d. determining, using a machine learning model, a score for each document of a set of the plurality of documents, wherein: i. the score for a document of the set is based on an angle between a vector of the plurality of vectors corresponding to the document of the set and the query vector and at least one weight, and ii. the at least one weight is determined based on one or more categories associated with the document of the set; e. ranking the set of the plurality of documents based on the score for each document of set; andf. providing the set of documents based on the ranking.

28. The computer-readable medium of claim 27, wherein the at least one weight is determined using a loss function.

29. The computer-readable medium of claim 27 or 28, wherein the one or more categories comprises one or more of quality, relevance, personalization, reputation, recency, popularity, or query item record.

30. The computer-readable medium of any one of 27 to 29, wherein the at least one weight is determined based on a metric associated with the ranking.

31. The computer-readable medium of any one of claims 27 to 30, wherein the metric comprises one or more of revenue, conversion, user engagement, or item history.

32. The computer-readable medium of any one of claims 27 to 31, wherein an angle between the query vector and a vector corresponding to a document is based on a cosine similarity of the query vector and the vector corresponding to the document.

33. The computer-readable medium of any one of claims 27 to 32, wherein the machine learning model comprises a transformer model.

34. The computer-readable medium of any one of claims 27 to 33, wherein the machine learning model is trained using a weighted cross entropy loss function.

35. The computer-readable medium of any one of claims 27 to 34, wherein each category of the one or more categories is associated with a category value.

36. The computer-readable medium of any one of claims 27 to 35, wherein a category of the one or more categories is prioritized when determining the at least one weight.

37. The computer-readable medium of any one of claims 27 to 36, wherein prioritizing the category of the one or more categories is based on the metric.

38. The computer-readable medium of any one of claims 27 to 37, further comprising providing the score for one or more of the set of documents.

39. The computer-readable medium of any one of claims 27 to 38, further comprising excluding one or more documents of the plurality of documents based on a threshold angle.

Images