Computer-implemented method and system for training neural network-based ordering model

Abstract

A computer-implemented method for training a neural network-based ranking model includes performing a training data enhancement operation on a training dataset to generate a synthetic training dataset, and training a neural network-based ranking model using the training dataset and the synthetic training dataset. For each query of the plurality of queries, the training dataset includes respective query document data and respective relevance determination data. Query-document data for a query includes data associated with a plurality of query-document pairs for the query. The relevance determination data for the query includes one or more sets of user feedback data associated with the query. The training dataset has an unbalanced training data distribution such that the amount of correlation determination data available for at least some of the plurality of queries differs, while the synthesized training dataset is adapted to reduce training data distribution imbalance for the training dataset. According to the embodiment of the invention, the performance of the tail query can be improved under the condition that the head query is not influenced.

Description

Technical Field

[0001] The present invention generally relates to the training and operation of neural network based ranking models. Background Art

[0002] Learning to rank (LTR) or machine learning to rank (MLR) is the application of machine learning techniques to generate ranking models for information systems. Different applications can generate different ranking models. The performance of these ranking models (such as accuracy, fairness, etc.) may depend on factors such as training data and training process. Summary of the invention

[0003] According to a first aspect of the present invention, a computer-implemented method for training a neural network-based ranking model is provided. The computer-implemented method comprises: performing a training data enhancement operation on a training data set to generate a synthetic training data set, and using the training data set and the synthetic training data set to train a neural network-based ranking model. For each of a plurality of queries, the training data set comprises respective query document data and respective relevance judgment data. The query-document data of the query comprises data associated with a plurality of query-document pairs of the query. The relevance judgment data of the query comprises one or more groups of user feedback (such as user clicks) data associated with the query. The training data set has an unbalanced training data distribution, so that the amount of available relevance judgment data for at least some of the plurality of queries is different, and the synthetic training data set is suitable for reducing the imbalance in the training data distribution of the training data set.

[0004] Optionally, the imbalanced training data distribution as a whole follows a long-tail distribution or a heavy-tail distribution.

[0005] Optionally, the training data enhancement operation includes: (i) for each query among multiple queries, determining a corresponding representation of the query; (ii) for each query among the multiple queries, determining one or more corresponding adjacent queries based on the determined representations of the multiple queries; and (iii) for one or more queries among the multiple queries, generating synthetic training data based on relevance judgment data associated with the query and relevance judgment data associated with one or more adjacent queries of the query.

[0006] Optionally, each query in the plurality of queries corresponds to a plurality of query-document pairs having corresponding features, and a corresponding representation of the query is determined according to the corresponding features of the query.

[0007] Optionally, the determination of the corresponding representation of the query is based on statistics of the corresponding features of the query.

[0008] Optionally, the determination of the corresponding representation of the query is based on an average value of the corresponding features of the query.

[0009] Optionally, the determination of one or more respective neighbor queries is based on a k-nearest neighbor (KNN) approach.

[0010] Optionally, generating synthetic training data for a query includes: (a) sampling a set of user feedback data associated with the query from multiple sets of user feedback data associated with the query to obtain a first data sample; (b) selecting one of the adjacent queries associated with the query and sampling a set of user feedback data associated with the selected adjacent query from multiple sets of user feedback data associated with the selected adjacent query to obtain a second data sample; and (c) synthesizing a data sample based on the first data sample and the second data sample. In some embodiments, the sampling in (a) may be random. In some embodiments, the selection in (b) may be random. In some embodiments, the sampling in (b) may be random.

[0011] Optionally, the synthesis of data samples is based on:

[0012]

[0013] Among them, l′ is a synthetic data sample, is the first data sample, is the second data sample, λ is a hyperparameter. Preferably, 0<λ<1.

[0014] Optionally, generating synthetic training data for one or more of the multiple queries further includes: repeating steps (a) to (c) to synthesize multiple data samples.

[0015] Optionally, the number of times steps (a) to (c) are repeated for each of the plurality of queries depends on (is adapted to) the amount of user feedback data associated with the query.

[0016] Optionally, the computer-implemented method further comprises: determining the occurrence frequency or relative occurrence frequency of each of the plurality of queries based on the relevance judgment data of the plurality of queries. Optionally, the number of times steps (a) to (c) are repeated for each of the plurality of queries depends on the determined occurrence frequency or relative occurrence frequency of the corresponding query.

[0017] Optionally, the relative frequency of occurrence of the query is determined based on:

[0018]

[0019] in, The tail metric corresponding to the query, t i is the number of sets of user feedback data for the query.

[0020] Optionally, the number of times steps (a) to (c) are repeated for each of the plurality of queries is proportional to the weighting factor w. i Associated, the weighting factor w i Defined as

[0021]

[0022] Among them, w e is a hyperparameter that controls the overall weight of synthetic data samples, T max and T min are the maximum and minimum values ​​of T, respectively, and w c is the threshold value.

[0023] Optionally, the neural network-based ranking model includes: a first model branch having a first multilayer perceptron and a first predictor operably coupled to the first multilayer perceptron; a second model branch having a second multilayer perceptron and a second predictor operably coupled to the second multilayer perceptron; and a combiner for combining an output of the first predictor and an output of the second predictor. The first multilayer perceptron and the second multilayer perceptron may have the same model architecture / structure. The first predictor and the second predictor may have the same architecture / structure.

[0024] Optionally, the combiner is configured to weight the output of the first predictor and / or the output of the second predictor.

[0025] Optionally, the training of the neural network-based ranking model includes: using the neural network-based ranking model to perform a ranking or scoring operation based on the training data set and the synthetic training data set.

[0026] Optionally, the ranking or scoring operation includes: processing the training dataset using the first model branch; and processing the combination of the training dataset and the synthesized training dataset using the second model branch.

[0027] Optionally, the computer-implemented method further comprises: determining a ranking or scoring loss according to the ranking or scoring operation performed

[0028] Optionally, the training of the neural network-based ranking model further includes: using the neural network-based ranking model to perform a comparative learning operation based on the training data set and / or the synthetic training data set.

[0029] Optionally, the contrastive learning operation includes: for one or more of the multiple queries, performing a data perturbation operation on the data associated with each of the multiple query-document pairs of the query to generate respective enhanced data for the data of each of the multiple query-document pairs; and processing the enhanced data using a neural network-based ranking model.

[0030] Optionally, the data perturbation operation includes: generating, for each of the data of each of the plurality of query-document pairs: a first set of enhanced data having a first noise injection degree; and a second set of enhanced data having a second noise injection degree different from the first noise injection degree.

[0031] Optionally, the neural network-based ranking model further includes: a first projector operably coupled to the first multilayer perceptron and a second projector operably coupled to the second multilayer perceptron. Optionally, the contrastive learning operation further includes: processing enhanced data (e.g., first and second sets of enhanced data) using the first multilayer perceptron and the first projector; and processing enhanced data (e.g., first and second sets of enhanced data) using the second multilayer perceptron and the second projector.

[0032] Optionally, the computer-implemented method further comprises: determining the contrast loss according to the execution of the contrast learning operation

[0033] Optionally, the computer-implemented method further comprises: performing a joint optimization operation based on the execution of the sorting or scoring operation and the execution of the contrastive learning operation.

[0034] Optionally, the joint optimization operation includes: jointly optimizing a sorting or scoring loss associated with performing a sorting or scoring operation and the contrastive loss associated with performing contrastive learning operations

[0035] Optionally, the joint optimization includes: optimizing the total loss It is equal to Where γ is a hyperparameter.

[0036] In a second aspect of the present invention, a system for training a neural network-based ranking model is provided, comprising: one or more processors, and a memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for executing or facilitating the execution of the computer-implemented method of the first aspect described above.

[0037] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which one or more programs are stored, which are configured to be executed by one or more processors, and the one or more programs include instructions for executing or facilitating the execution of the computer-implemented method of the first aspect mentioned above.

[0038] In a fourth aspect of the invention, there is provided a computer-implemented method for operating a neural network-based ranking model, comprising: processing a query and a document data set (data associated with a plurality of documents) using the neural network-based ranking model trained by the computer-implemented method of the first aspect above to determine a result. Optionally, the computer-implemented method further comprises presenting (e.g., displaying) the result. The result may include a ranked or ordered list of documents.

[0039] In a fifth aspect of the present invention, a system for training a neural network-based ranking model is provided, the system comprising: one or more processors, and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising instructions for executing or facilitating the execution of the computer-implemented method of the fourth aspect above.

[0040] In a sixth aspect of the present invention, a non-temporary computer-readable storage medium is provided, in which one or more programs are stored, which are configured to be executed by one or more processors, and the one or more programs include instructions for executing or facilitating the execution of the computer-implemented method of the fourth aspect above.

[0041] Other features and aspects of the invention will become apparent from consideration of the detailed description and accompanying drawings.Any feature described herein in relation to one aspect or embodiment may be used in combination with any other feature described herein in relation to any other aspect or embodiment.

[0042] The term "document" as used herein, unless otherwise specified, generally refers to any information item, such as a digital image, photograph, electronic document or file, email message, voice mail message, short message service message, web page, portion of a web page, map, electronic link, commercial product, multimedia file, song, book, album, article, database record, abstract of any one or more of the above items, etc. Such information can be retrieved by querying a server (such as a search engine).

[0043] Depending on the context, terms such as "generally," "about," "substantially," or similar terms of degree are used to account for manufacturing tolerances, degradation, trends, tendencies, imperfect physical conditions, and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings:

[0045] Figure 1 is a schematic diagram illustrating a system including a query server and a plurality of user devices in one example.

[0046] Figure 2The following is a diagram that illustrates the query operation in an example.

[0047] Figure 3A The following is a diagram of the training of a neural network-based sorting model in an example.

[0048] Figure 3B yes Figure 3A Schematic diagram of the test of the neural network-based ranking model trained in .

[0049] Figure 4 is a flowchart of a computer-implemented method for training a neural network-based ranking model in accordance with certain embodiments of the present invention.

[0050] Figure 5 Schematic diagram illustrating the training data enhancement operation in some embodiments of the present invention.

[0051] Figure 6 is a flowchart illustrating a method for training data enhancement in certain embodiments of the present invention.

[0052] Figure 7 is a flow chart of a method for generating synthetic training data in certain embodiments of the present invention;

[0053] Figure 8 is a schematic diagram of using a neural network-based ranking model to perform ranking or scoring operations in certain embodiments of the present invention.

[0054] Fig. 9 is a schematic diagram of a data perturbation operation in some embodiments of the present invention.

[0055] Fig.10 is a schematic diagram of a comparative learning operation in certain embodiments of the present invention.

[0056] Fig.11 is a flow chart of a joint optimization method in some embodiments of the present invention.

[0057] Fig.12 It is a data enhancement and contrastive learning framework long-tail learning ranking (DCLR) framework in one embodiment of the present invention.

[0058] Fig.13 Yes Display Fig.12 Table showing the performance of the DCLR framework and other methods on two different datasets (Tiangong-ULTR dataset and Istella-S dataset).

[0059] Fig.14 yes Fig.12 Table of ablation study results of the DCLR framework on the Tiangong-ULTR dataset.

[0060] Fig.15It is a block diagram of a data processing system in some embodiments of the present invention. The system can be used to partially or completely execute one or more methods of the present invention. DETAILED DESCRIPTION

[0061] Figure 1 An example system 100 is shown, in which there are N user devices 10-1 to 10-N (N can be any integer), which are operably connected to a query server 20. Each user device 10-1 to 10-N is configured to communicate with the query server 20 via one or more communication links, which can be wired or wireless. The user devices 10-1 to 10-N are configured to send queries to the query server 20. The query server 20 is used to receive and process the queries, and then feed back the query results to the user devices 10-1 to 10-N. The query may be related to a search, and the query results may include multiple documents determined by the query server 20 to be most relevant to the search.

[0062] Figure 2 An example of a query operation 200 performed by a ranking system 202 is shown. The ranking system 202 includes a ranking or scoring model, which may be a neural network-based model, for performing ranking or scoring tasks to rank or score candidate documents associated with the query. In this example, the ranking system 202 is operably connected to a document library 204, wherein the document library 204 stores information related to the candidate documents. The ranking system is configured to retrieve or receive information related to the candidate documents from the document library 204. In operation, the ranking system 202 receives a query, such as a query from a user device, and retrieves or receives information related to the candidate documents from the document library 204. The ranking system 202 then processes the query and information using the ranking or scoring model, generates results, and then provides them to the user device that submitted the query. The ranking or scoring model may provide respective rankings or scores for some or all of the candidate documents. The results generated by the ranking system 202 may include a list of candidate documents, which is ranked according to the determined relevance of the documents to the query. For example, the candidate documents may be ranked in ascending or descending order, and one or more results may be filtered or not filtered. In some operations, a user on a user device may respond to the results and select one or more documents from the list. For example, a user may provide user feedback or input (such as a click) on a user device to select one or more documents from the list. The user feedback or input indicates that the user may think that the selected one or more documents are more relevant to the query than other documents. The user feedback or input may be received by the query server and stored for subsequent use in training the model of the ranking system 202.

[0063] Figure 3A Demonstrated the ability to generate or train neural network-based ranking or scoring models (e.g. Figure 2The process includes providing training data to a learning system for processing. The learning system generates or optimizes a ranking or scoring model based on the training data. In this example, the training data may include multiple queries, multiple documents for the queries (e.g., each query may include multiple documents), and ground truth. The ground truth indicates the judged relevance (e.g., score, ranking, etc.) of one or more documents to the corresponding query. The judged relevance can be obtained from user feedback (e.g., user input, user behavior, etc.), for example, a response to the query result detected or received on a user device.

[0064] Figure 3B Demonstrated testing Figure 3A The general process of using a trained ranking or scoring model in a ranking or scoring model. The process includes providing test data to the trained ranking or scoring model for processing. In this example, the test data may be in the same format as the training data, except that ground truth is not provided to the trained ranking or scoring model. The trained ranking or scoring model is configured to process the test data (queries and documents) and generate results corresponding to the queries. For example, each result includes documents ranked according to their predicted relevance to the query. The ground truth of the test data is used to compare with the results generated by the trained ranking or scoring model to determine its performance. If the performance of the trained ranking or scoring model is determined to be insufficient, the trained ranking or scoring model can be further trained.

[0065] Figure 4 A general computer-implemented method 400 for training a neural network-based ranking model in some embodiments of the present invention is shown. The method 400 includes, in step 402, performing a training data enhancement operation on a training data set to generate a synthetic training data set. For each query in a plurality of queries, the training data set includes its corresponding query document data and corresponding relevance judgment data. The query-document data of the query includes data associated with a plurality of query-document pairs of the query. The relevance judgment data of the query includes one or more sets of user feedback (such as user clicks) data related to the query. The training data set has an unbalanced training data distribution, so that the amount of relevance judgment data of at least some queries is different. The synthetic training data set generated in step 402 is configured to reduce the imbalance of the training data distribution of the training data set. In some embodiments, the unbalanced training data distribution of the training data set generally follows a long-tail distribution or a heavy-tail distribution. In step 404, the method 400 also includes training the neural network-based ranking model using the training data set and the synthetic training data set. The training of the neural network-based ranking model in step 404 may include various operations, which will be further discussed below.

[0066] Figure 5The training data enhancement operation in some embodiments of the present invention is shown. In some embodiments, Figure 5 The training data augmentation operation can be regarded as belonging to Figure 4 The training data enhancement operation in step 402. For ease of description, unless otherwise specified, the following Figure 5 The description of the training data enhancement operation will refer to Figure 4 Step 402.

[0067] like Figure 5 As shown, the training data is processed by the training data enhancement system 500 to generate synthetic training data. The training data and the synthetic training data can be reference Figure 4 The data described by the training data enhancement operation in step 402. Specifically, the training data has an unbalanced or skewed data distribution, and the synthetic training data is configured to reduce such unbalanced or skewed data distribution (make the data distribution more balanced or less skewed). In some embodiments, the training data enhancement system 500 is also configured to receive data associated with the required degree of data enhancement, and is configured to generate synthetic training data according to the required degree of data enhancement. For example, the required degree of data enhancement may affect the amount of synthetic training data generated. If the range of required data enhancement is smaller, the amount of synthetic training data generated is smaller, and vice versa. The required degree of data enhancement is related to the degree of reducing the unbalanced or skewed data distribution of the training data. The required degree of data enhancement can be adjusted as needed.

[0068] Figure 6 600 is a method for enhancing training data in some embodiments of the present invention. In some embodiments, the method 600 may be considered as belonging to Figure 4 The training data enhancement operation in step 402. For ease of description, unless otherwise specified, the following Figure 6 The description of method 600 will refer to Figure 4 Step 402.

[0069] Method 600 includes determining a respective representation for each query in step 602. In some embodiments, each query may correspond to a plurality of query-document pairs having respective features (e.g., feature vectors), and the determination of the representation of each respective query is based on the respective features of the query. For example, the representation of each respective query is determined based on a statistic (e.g., an average) of the respective features of the query.

[0070] The method 600 further includes determining one or more corresponding neighboring queries for each query based on the determined representation of the query in step 604. In some embodiments, a KNN method may be used to perform such determination.

[0071] The method 600 further includes, in step 606, generating synthetic training data for one or more queries based on the relevance judgment data associated with the query and the relevance judgment data associated with one or more neighboring queries of the query. In some embodiments, synthetic training data is generated only for one or some queries to solve the data distribution imbalance problem mentioned above.

[0072] Figure 7 700 of generating synthetic training data in some embodiments of the present invention. In some embodiments, the method 700 can be viewed as Figure 6 Example implementation of step 606 for each query in method 600. Figure 6 The method 700 can be used repeatedly for different queries in the method 600. For ease of presentation, unless otherwise specified, the following Figure 7 The description of method 700 will refer to Figure 6 Step 606.

[0073] The method 700 includes sampling a set of user feedback data associated with the query from multiple sets of user feedback data associated with the query to obtain a first data sample in step 702. The sampling in step 702 may be random or pseudo-random.

[0074] The method 700 further includes, in step 704, selecting one of the neighboring queries associated with the query, and sampling a set of user feedback data associated with the selected neighboring query from the multiple sets of user feedback data associated with the selected neighboring query to obtain a second data sample. The selection in step 704 may be random or pseudo-random. The sampling in step 704 may be random or pseudo-random.

[0075] Steps 702 and 704 may be performed in any order or simultaneously.

[0076] The method 700 further includes, in step 706 after steps 702 and 704, synthesizing a data sample based on the first data sample and the second data sample. In some embodiments, the synthesis of the data sample is based on the following conditions:

[0077]

[0078] Among them, l′ is a synthetic data sample, is the first data sample, is the second data sample, and λ is a hyperparameter. It is best if 0<λ<1, so that the operation does not correspond to data resampling.

[0079] Method 700 also includes, in step 708, determining whether sufficient data samples have been synthesized for the query. If it is determined that a sufficient number of data samples have been synthesized for the query, method 700 ends. If it is determined that more data samples need to be synthesized for the query, method 700 returns to step 702 (or step 704, as steps 702 and 704 can be performed in any order or simultaneously). In some embodiments, the number of repetitions of steps 702 to 706 for each query depends on the amount of available user feedback data related to the query and is therefore adapted thereto.

[0080] In some embodiments, method 700 further includes determining the occurrence frequency or relative occurrence frequency of each query based on the query relevance determination data, and the number of repetitions of steps 702 to 706 for each query depends on the determined occurrence frequency or relative occurrence frequency of the corresponding query. In some embodiments, the relative occurrence frequency of the query is determined based on the following conditions:

[0081]

[0082] in, corresponds to the tail metric of the query, t i is the number of sets of user feedback data for the query. In some embodiments, the number of repetitions of steps 702 to 706 for each query is proportional to the weighting factor w i Associated, the weighting factor w i Defined as

[0083]

[0084] Among them, w e is a hyperparameter that controls the overall weight of synthetic data samples, T max and T min are the maximum and minimum values ​​of T, respectively, and w c is the threshold value.

[0085] Figure 8 The ranking or scoring operation performed using the neural network-based ranking model 800 in some embodiments of the present invention is shown. In some embodiments, Figure 8 The sorting or scoring operation can be regarded as belonging to Figure 4 Training operation in step 404. The training of the neural network-based ranking model in step 404 may include using the neural network-based ranking model to perform ranking or scoring operations (such as Figure 8 For ease of presentation, unless otherwise stated, the following descriptions of Figure 8 Descriptions of sorting or scoring operations will refer to Figure 4 Step 404.

[0086] like Figure 8 As shown, in some embodiments, the neural network-based ranking model includes multiple model branches. In some embodiments, the neural network-based ranking model includes a first model branch having a multilayer perceptron MLP1 and a predictor P1 operably coupled to the multilayer perceptron MLP1, a second model branch having a multilayer perceptron MLP2 and a predictor P2 operably coupled to the multilayer perceptron MLP2, and a combiner for combining the outputs of predictors P1 and P2. In some examples, the multilayer perceptrons MLP1 and MLP2 may have the same model architecture / structure. In some examples, predictors P1 and P2 may have the same architecture / structure. In some embodiments, the combiner is arranged to apply a weight w1 to the output of predictor P1 and / or a weight w2 to the output of predictor P2.

[0087] like Figure 8 As shown, the ranking or scoring operation includes processing the training data set with a first model branch, and processing the training data set and the synthetic training data set (e.g., based on Figures 5 to 7 In some embodiments, the ranking or scoring loss can be determined based on the ranking or scoring operation performed.

[0088] Fig. 9 The data perturbation operation in some embodiments of the present invention is shown. The data perturbation operation is configured to facilitate the contrastive learning operation. Fig. 9 As shown, the data perturbation system 900 processes the query-document pair p i To generate an enhanced query-document pair. In some embodiments, the enhanced query-document pair includes two enhanced query-document pairs p′ i and p″ i The data perturbation operation may be performed on one or more query-document pairs of a query and on one or more queries. In some embodiments, the data perturbation system 900 is configured to provide a query-document pair p i Add noise to generate enhanced query-document pairs.

[0089] Fig.10 The contrastive learning operation in some embodiments of the present invention is shown. In some embodiments, Fig.10 The contrastive learning operation in Figure 4 Training in step 404. The training of the neural network-based ranking model in step 404 may include using the neural network-based ranking model to perform a contrastive learning operation (e.g., Fig.10 For ease of presentation, unless otherwise stated, the following descriptions of Fig.10The description of the contrastive learning operation will refer to Figure 4 Step 404.

[0090] Although in Fig.10 , but in some embodiments, the contrastive learning operation includes: for one or more queries, for each query-document pair associated with the query, performing a data perturbation operation to generate enhanced data for each query-document pair. The data perturbation operation may be Fig. 9 The contrastive learning operation can be performed on one or more queries.

[0091] In some embodiments, the contrastive learning operation includes processing the augmented data using a neural network-based ranking model. Fig.10 As shown, the neural network-based sorting model includes: a first branch with a multilayer perceptron MLP1 and a projector PJ1, and a second branch with a multilayer perceptron MLP2 and a projector PJ2. In some embodiments, the multilayer perceptrons MLP1 and MLP2 in the model 1000 may be Figure 8 In some examples, the multilayer perceptrons MLP1 and MLP2 in model 800 may have the same model architecture / structure. In some examples, the projectors PJ1 and PJ2 may have the same architecture / structure. In some examples, the projectors PJ1 and PJ2 may be multilayer perceptrons.

[0092] like Fig.10 As shown, the contrastive learning operation may include processing the enhanced query-document pair p′ using the first branch and the second branch of the model 1000, respectively. i and p″ i (from Fig. 9 Specifically, the processing may include (1) using a multilayer perceptron MLP1 to process the enhanced query-document pair p′ i and p″ i , to obtain the potential embedded h′ i and h″ i , and uses projector PJ1 to process the potential embedded h′ i and h″ i , to obtain the projection / embedded g′ i and g″ i ; (2) Use multi-layer perceptron MLP2 to process the enhanced query-document pair p′ i and p″ i , to obtain the potential embedded h′ i and h″ i , and uses projector PJ2 to process the potential embedded h′ i and h″ i, to obtain the projection / embedded g′ i and g″ i In some embodiments, the contrast loss According to the projection / embedded g′ i and g″ i to be determined by the implementation of the

[0093] In certain embodiments, Figure 8 Model 800 and Fig.10 The models 1000 may be combined or integrated.

[0094] Fig.11 FIG. 1 shows a joint optimization method 1100 in some embodiments of the present invention. In some embodiments, the method 1100 may be based on Figure 8 Sorting or scoring operations in Fig.10 It is performed by the contrastive learning operation in .

[0095] Method 1100 includes, at step 1102A, based on a contrastive learning operation (e.g. Fig.10 Contrastive learning operation in ) determines the contrastive loss And in step 1102B, based on the sorting or scoring operation (e.g. Figure 8 The sorting or scoring operation in Steps 1102A and 1102B may be performed in any order or simultaneously.

[0096] The method 1100 also includes jointly optimizing the determined ranking loss in step 1104 and contrast loss In some embodiments, this may include optimizing the models 800, 1000, in particular the multilayer perceptrons MLP1, MLP2 in the models 800, 1000. In some embodiments, step 1104 includes optimizing the total loss This loss is equal to where γ is a hyperparameter.

[0097] Combine the following Figure 12 to Figure 14 An example illustrating a neural network-based ranking model training technique.

[0098] In existing search engine systems, learning to rank (LTR) is to learn a model (ranker) from user click data and return the order of a list of candidate documents. According to the article "A large scale search dataset for unbiased learning to rank" (2022) by Zou et al., user click behavior may follow a long-tail distribution. In other words, some more popular queries have many user clicks (i.e., head queries). In LTR, these queries may perform better than queries with fewer clicks (i.e., tail queries) because data imbalance may cause the ranker to pay more attention to the head. Therefore, this may cause unfairness to tail queries. This problem can be called long-tail LTR. The following example aims to solve this problem to improve tail query performance.

[0099] One embodiment of the present invention provides a data enhancement and contrastive learning framework long-tail LTR (also referred to as "DCLR"). The DCLR in the embodiment of the present invention adopts a bilateral branch network, which can effectively learn from the head and tail of the data distribution, thereby overcoming the difficulty of learning from an unbalanced training data distribution. The DCLR of this embodiment uses an adaptive data enhancement module to synthesize new data, which helps to alleviate the problem of scarce tail data. The DCLR of this embodiment also combines contrastive learning to learn a more uniform distribution for tail queries by creating multiple (for example, two) enhanced views and maximizing their consistency. The DCLR of this embodiment also utilizes a multi-task training strategy to jointly optimize the model.

[0100] In the following disclosure related to the DCLR embodiment, the long-tail LTR problem is studied and a DCLR design embodiment is introduced. The DCLR embodiment uses a bilateral branch network to learn from different training data distributions and uses an adaptive data enhancement module to change the training data distribution to alleviate the data sparsity problem. The DCLR implementation scheme also uses a contrastive learning module in the long-tail LTR to learn a more unified representation through contrastive tasks.

[0101] In the description Fig.12 Before describing the framework 1200 in the present embodiment, we first introduce some preliminary knowledge applicable to this embodiment. Given a set of queries Q = {q1, ..., q p}, each query q k ∈Q and n k documents. Assume that many users browse these queries and click on the documents they are interested in. The cumulative number of queries in all user click sessions will produce a long-tail distribution. One of the goals of framework 1200 in this embodiment is to improve the performance of tail queries (and the average performance of all queries).

[0102] like Fig.12As shown, the framework 1200 in this embodiment consists of an LTR task and a contrast task. The LTR task includes a bilateral branch network structure and an adaptive data enhancement module. The contrast task includes temporary data perturbations and contrast losses. A multi-task training strategy is used to perform joint optimization. Please note that the framework 1200 in this embodiment is independent of the model of the neural network-based learning ranking model (also called a "neural sorter").

[0103] Fig.12 A bilateral branch network module / structure 1202 in one embodiment is shown, which can be used to adjust the weight of the long tail portion, thereby alleviating the long tail problem. Specifically, the two branches have the same network structure, but different input data. The input data is directly processed in one branch of the neural network-based learning ranking model, and adaptive enhancement is performed before processing in one branch of the neural network-based learning ranking model. The details of the adaptive enhancement are as follows. After passing through the multi-layer perception layer (MLP) and prediction layer of the bilateral branch network structure 1202, the outputs of the two branch networks (prediction layers) are combined according to the following formula:

[0104] s=αs1+(1-α)s2 (1)

[0105] Among them, s is the final output prediction score of the LTR task, s1 and s2 are the prediction scores of each branch, and α is a hyperparameter that controls the proportion of each branch.

[0106] Fig.12 An adaptive training data enhancement module 1204 in one embodiment is also shown. Since the training data is highly skewed, a data enhancement operation needs to be performed to synthesize new data and change the distribution of the training data. In this embodiment, a representation of each query is first obtained. Assume that a query qi corresponds to a set of query document pairs { i ,d1>,…, i ,d n >}, corresponding to the feature {f i,1 ,…,f i,n In this embodiment, q i It is expressed as the average value of these features, that is, Then, the semantic neighborhood of each query is obtained. In this embodiment, the kNN algorithm is applied to find the query, q i k nearest neighbors in the semantic space. k nearest neighbors can be simplified as {q i,1 ,…,q i,k Then, the training data is synthesized based on the existing training data. In one embodiment, the user clicks on the query q i The sessions (i.e., query-document pair lists and corresponding user click feedback) are randomly sampled and recorded as At the same time, randomly select query q​​j ∈{q i,1 ,…,q i,k} and sample the corresponding user click sessions Then, in this embodiment, the two data samples are mixed according to the following principles:

[0107]

[0108] Where l′ is the synthetic data sample and λ is a hyperparameter that controls the ratio of the two data samples. Note that when λ=0 or λ=1, this is the same as data resampling. It is best if λ is not equal to 0 or 1.

[0109] In this embodiment, an adaptive sampling strategy is also adopted. First, the tail degree is defined, which describes the degree to which the query is located in the tail. Assume that the query q i Appears in all user click sessions i times. Then query q i The tail degree (with means) is defined as Assume that w i is the control query q i The weight of data enhancement (i.e., the number of synthetic samples) is defined as

[0110]

[0111] Among them, w e is a hyperparameter that controls the overall weight of the added data sample, T max and T min is the maximum and minimum value of T, w c is the cutoff value.

[0112] Fig.12 Also shown is a contrastive learning module 1206 in one embodiment. In this embodiment, contrastive learning is deployed in the long-tail LTR to learn better representations. According to the standard paradigm of contrastive learning, two enhanced views are constructed for the input data. The enhanced views are then passed to the neural network MLP and processed to obtain two latent vectors. Contrastive loss is used to minimize the distance between the two representations.

[0113] The inventors of the present invention believe that data enhancement is very important in contrastive learning, and different situations or applications may require different data enhancement methods. The present invention adopts a task-oriented long-tail LTR enhancement method to perturb the input data and promote the representation learning process.

[0114] In this embodiment, given a set of query-document pairs P, for each query-document pair p with order dimension features i∈P, perform data enhancement and generate two enhanced views. This embodiment uses random noise injection method to perturb the input data. The specific process is as follows

[0115]

[0116] where Δ′ i and Δ″ i Each element of is uniformly sampled from [0,1]. ∈ is a small positive constant. This ensures that adding noise to the input data does not lead to large deviations.

[0117] In this embodiment, the neural network-based learning ranking model uses a multi-layer perception (MLP) to process the input data and obtain a latent embedding, and then uses a prediction layer to predict the output score based on the latent embedding. Therefore, in the neural network-based learning ranking model, after random perturbations, the two inputs are passed through layers before the final output layer to obtain the latent embedding h′ i and h″ i , recorded as:

[0118] h′ i =MLP(p′ i ),h″ i =MLP(p″) i ) (5)

[0119] Next, a nonlinear projector is used to project the latent embedding, denoted as:

[0120] g′ i =Project(h′ i ),g″ i =Project(h″ i ) (6)

[0121] In this embodiment, the projector is implanted through another tower-shaped MLP. According to the contrastive learning paradigm, g′ i and g″ i The consistency is maximized.

[0122] The framework 1200 of this embodiment also uses the contrast loss InfoNCE disclosed by M. Gutmann et al. in the paper "Noise-contrastiveestimation: A new estimation principle for unnormalized statistical models" to maximize the consistency of positive comparisons and minimize the consistency of negative comparisons. Assume g′ i g″ i is the embedding of the two views generated by random perturbations, then the contrast loss It is expressed as:

[0123]

[0124] Among them, s() measures the similarity between two vectors and is set as the cosine similarity function. τ is a hyperparameter, which is called temperature in softmax. In this way, representation learning can be enhanced and model training can be promoted.

[0125] The framework 1200 of this embodiment uses a multi-task training strategy to jointly optimize the learning to rank loss and the contrast loss. The learning to rank loss (expressed as ) varies with the type of neural network-based learning ranking model. In this embodiment, the overall loss is expressed as:

[0126]

[0127] Here, γ is a hyperparameter that controls the strength of the contrast loss.

[0128] For evaluation Fig.12 In order to test the performance of the DCLR framework in

[15] , experiments were conducted. A real-world dataset called Tiangong-ULTR (available from http: / / www.thuir.cn / data-tiangong-ultr / ) was used in the experiment, which contains real user click behaviors. The data was preprocessed. Specifically, sessions without user clicks were first filtered out, and then 457 queries, 7743 documents, and 46412 valid click sessions were randomly extracted from the dataset. Each click session contains 10 documents. The feature dimension of each query-document pair is 33.

[0129] User clicks are simulated using another dataset, Istella-S, which is published in the paper "Post-learning optimization of tree ensembles for efficient ranking" by C. Lucchese et al., and the methods used are published in the paper "Unbiased Learning to Rank with Unbiased Propensity Estimation" by Q. Ai et al. and the paper "Model-based unbiased learning to rank" by D. Luo et al. 788 queries, 66,143 documents, and 46,994 valid click sessions were sampled. Each document-query pair is represented by 220-dimensional features. The cascade model published in the paper "An experimental comparison of click position-bias models" by N. Craswell et al. is used to generate test probabilities, which assumes that users decide whether to click on each result before moving to the next result and stop after the first click. Pareto distribution is used to simulate user clicks on each query result.

[0130] Some existing models were also tested to compare Fig.12 The DCLR framework 1200 in the embodiment is compared:

[0131] LambdaRank as described in Burge’s From ranknet to lambdarank to lambdamart.

[0132] Resampling strategies - oversampling and undersampling, disclosed in Jason Brownlee's "Imbalanced classification with Python: better metrics, balance skewed classes, cost-sensitive learning". In short, oversampling samples more from the tail queries to rebalance the head and tail queries in the training data distribution; while in contrast to oversampling, undersampling keeps the tail queries unchanged and reduces the sampling of the head queries.

[0133] Cui et al. published a method to improve the sample loss function, ClassBalance, in the paper "Classbalanced loss based on effective number of samples". ClassBalance calculates the effective number of samples and reweights the loss for each class.

[0134] For examples of curriculum learning strategies, see Zhang et al., Head2tail and Tail2head. Head2tail trains a neural network-based learning-to-rank model using head queries in the first curriculum phase, and then fine-tunes the ranker using only tail queries in the second curriculum phase. Tail2head takes the opposite steps, training on tail queries first and then fine-tuning on head queries.

[0135] To evaluate the performance of all methods, the commonly used Normalized Discounted Cumulative Gain (NDCG) was used in the experiments (see et al., "Cumulated gain-based evaluation of IR techniques") and expected reciprocal rank (ERR) (see Chapelle et al., "Expected reciprocal rank for graded relevance") are used as metrics. For both metrics, the results of rank 1 and rank 3 are reported to show the performance of the model at different positions.

[0136] The Pareto principle (published in Box et al., An analysis for unreplicated fractional factorials) is used as the criterion for dividing head and tail queries. In this experiment, the queries with the top 20% of the number of occurrences are set as head queries, and the rest are set as tail queries. Evaluation metrics for tail and head query sets are reported. In this experiment, the training set, validation set, and test set are randomly split in a ratio of 7:1:2. Cross-validation is used to select the best hyperparameters. The LTR model uses a two-layer MLP (MLP1, MLP2) with 32 and 16 nodes in the first and second layers, respectively. The projector uses another two-layer MLP (PJ1, PJ2) with 16 and 8 nodes, respectively. In this experiment: the batch size is 256, τ is 0.15, γ is 0.1, λ is set to 0.5, and ε is 0.2. For fair comparison, these settings remain unchanged in all experiments unless otherwise specified.

[0137] Fig.12 The DCLR model in the embodiment of the framework 1200 and various existing models are evaluated on two datasets to explore the performance of the models in different scenarios.

[0138] Table 1 summarizes the LTR performance for all queries, head queries, and tail queries on these datasets. It can be seen that DCLR generally improves on both tail and head queries compared to LambdaRank. This suggests that it is important to consider contrastive learning and training data augmentation for queries with long-tail distributions. The results show that Framework 1200 can benefit the long-tail LTR problem. It can also be seen that DCLR outperforms the reweighting methods (i.e., oversampling, undersampling) on ​​all three splits. Among these methods, oversampling is generally inferior to undersampling. However, resampling methods may change the distribution of the original data, which may have a negative impact on the overall performance of the model. For the refinement loss function strategy, the possible reason is that its trade-off between head terms and item terms is not healthy or may be harmful. It can also be seen that DCLR performs better than the curriculum learning-based model. The results show that combining data perturbation with contrastive learning can learn more unified and robust representations. This will be beneficial to improving the performance of the model.

[0139] exist Fig.12 In the framework 1200 of , an ablation study is also performed to analyze the different components of the DCLR framework 1200. Table 2 shows the results of the ablation study. It can be seen that the full model outperforms all variants on the three segmentations, which shows that all major components contribute to the improvement in performance.

[0140] One of the ablation studies was conducted without In this variant, the contrast loss Removed, only LTR loss is used to train the model. As can be seen, there is no contrast loss The DCLR performance is worse. This means The representation can be effectively enhanced to learn more unified and robust latent embeddings, which can improve the performance of the model.

[0141] Another ablation study is conducted without adaptive data augmentation: In this variant, adaptive data augmentation is replaced by uniform data augmentation (other parts remain unchanged). It can be seen that the model without adaptive data augmentation performs worse on tail queries, which indicates that adaptive data augmentation is beneficial to the performance of tail queries because it can retain more critical information for tail queries.

[0142] Another ablation study is performed without the bilateral branch network: In this variant, the bilateral branch network is removed (other components remain unchanged). It can be seen that without the bilateral branch network, the performance of tail queries degrades. This suggests that the combination of the two branches is beneficial for improving long-tail performance.

[0143] The DCLR implementation scheme of the present invention provides a framework based on data enhancement and contrastive learning to solve the long-tail LTR problem. DCLR uses a bilateral branch network to dynamically adjust the distribution of training data, uses a training data enhancement module to improve the generalization ability of the model by synthesizing data, and uses a contrastive learning module to improve the representation learning ability by utilizing the contrast signal of the feature.

[0144] Embodiments of the present invention may provide one or more of the following advantages. For example, some embodiments of the present invention solve the long-tail learning to rank problem. Some embodiments of the present invention are specifically designed to solve the long-tail learning to rank problem, which cannot be effectively solved by existing technologies. By improving the performance of tail queries without affecting the performance of head queries, the algorithm can provide a fairer and more balanced ranking system. For example, some embodiments of the present invention utilize data augmentation and contrastive learning. For example, some embodiments of the present invention use a combination of data augmentation and contrastive learning techniques to improve the generalization ability of the model to unseen examples and learn more unified and robust representations. This combination of techniques can lead to more efficient learning and better performance. For example, some embodiments of the present invention use dynamic weighting for head and tail queries. For example, some embodiments of the present invention provide a bilateral branch network module that dynamically adjusts the weights of head and tail queries. This means that the model in some embodiments can adapt to different data distributions and provide better performance for head and tail queries. For example, some embodiments of the present invention can be used in a variety of scenarios. For example, some embodiments of the present invention can be applied to various fields that use learning to rank algorithms, including search engines, e-commerce, advertising, and recommendation systems. This means that the algorithm has a wide range of applicability and can work in many different environments. For example, some embodiments of the present invention provide a more efficient and fairer ranking system by solving the long-tail learning ranking problem, using novel data enhancement and contrastive learning techniques, and adapting to different data distributions. These advantages make the algorithms of these embodiments a promising solution for various scenarios where learning to rank is important. Embodiments of the present invention may provide one or more additional or alternative advantages not specifically described.

[0145] Some embodiments of the present invention effectively solve the long-tail learning to rank problem in learning to rank scenarios. The long-tail learning to rank problem refers to the performance imbalance between head queries (i.e., queries with a large number of user clicks) and tail queries (i.e., queries with fewer user clicks), which creates an unfair situation for the latter. Some embodiments of the present invention provide a data enhancement and contrastive learning method, such as a DCLR model, to solve this problem. Some embodiments of the present invention provide an algorithm that uses a bilateral branch network module to dynamically adjust the weights of head and tail queries, uses an adaptive training data enhancement module to synthesize data and modify the training data distribution, and uses contrastive learning to learn a more unified and robust representation.

[0146] Embodiments of the present invention can be applied to various fields using learning to sort algorithms. These fields include:

[0147] Search engines: The algorithm can be used to improve the ranking of less common search queries, thereby enhancing the user experience and increasing click-through rates.

[0148] E-commerce: The algorithm can be used to improve recommendations for less popular products, thereby increasing sales and customer satisfaction.

[0149] Advertising: The algorithm can be used to improve ad targeting for less popular keywords, thereby increasing click-through rates and ROI.

[0150] Recommender systems: The algorithm can be used to improve recommendations for less popular items, leading to better user experience and higher engagement.

[0151] More generally, embodiments of the present invention can be applied to any scenario where there is a training data imbalance problem in learning to rank, and can improve the performance of tail queries without affecting head queries.

[0152] Fig.15An example data processing system 1500 is shown, which in one embodiment of the invention may be used as a server (e.g., query server 20), a user device (user device 10-N), or other type of data processing system. Data processing system 1500 generally includes appropriate components required to receive, store, and execute appropriate computer instructions, commands, and / or codes. The main components of data processing system 1500 are processor 1502 and memory (storage) 1504. Processor 1502 may include one or more: CPU, MCU, GPU, logic circuit, RaspberryPi chip, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or any other digital or analog circuit / circuitry that is configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. Processor 1502 may be arranged to perform data processing using machine learning-based techniques and using non-machine learning-based techniques. The memory 1504 may include one or more volatile memories (e.g., RAM, DRAM, SRAM, etc.), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or any combination thereof. Appropriate computer instructions, commands, codes, information, and / or data may be stored in the memory 1504. Computer instructions for executing or facilitating the execution of one or more method embodiments of the present invention may be stored in the memory 1504. For example, a neural network-based model of an embodiment of the present invention may be stored in the memory 1504. For example, training / testing / validation data of a neural network-based model of an embodiment of the present invention may be stored in the memory 1504. The processor 1502 and the memory (storage) 1504 may be integrated or separated (and operably connected). Optionally, the data processing system 1500 further includes one or more input devices 1506. Examples of such input devices 1506 include: keyboard, mouse, stylus, image scanner, microphone, tactile / touch input device (e.g., touch-sensitive screen), image / video input device (e.g., camera), etc. Optionally, the data processing system 1500 also includes one or more output devices 1508. Examples of such output devices 1508 include: displays (such as monitors, screens, projectors, etc.), speakers, headphones, earphones, printers, additive manufacturing machines (such as 3D printers), etc. The display screen may include an LCD display screen, an LED / OLED display screen, or other suitable display screens, which may be touch-sensitive or non-touch-sensitive. The data processing system 1500 may further include one or more disk drives 1512, which may include one or more of a solid-state drive, a hard drive, an optical drive, a flash drive, a tape drive, etc.A suitable operating system may be installed in the data processing system 1500, for example, on a disk drive 1512 or in the memory 1504. The memory 1504 and the disk drive 1512 may be operated by the processor 1502. Optionally, the data processing system 1500 further includes a communication device 1510 for establishing one or more communication links (not shown) with one or more other computing devices (such as a server, a personal computer, a terminal, a tablet computer, a mobile phone, a watch, an Internet of Things device, or other wireless computing devices). The communication device 1510 may include one or more of the following devices: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, etc. Transceiver, radio frequency transceiver, cellular (2G, 3G, 4G, 5G, 5G or above or similar) transceiver, optical port, infrared port, USB connection or other wired or wireless communication interface. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, independent transmitter and receiver, etc.). The communication link can be wired or wireless and is used to communicate commands, instructions, information and / or data. In one example, the processor 1502, memory 1504 (optionally input device 1506, output device 1508, communication device 1510 and disk drive 1512 (if any)) are directly or indirectly connected to each other via a bus, peripheral component interconnect (PCI) (such as PCI Express), universal serial bus (USB), optical bus or other similar bus structure. In one embodiment, at least some of these components can be connected wirelessly, for example through a network, such as the Internet or a cloud computing network. It will be appreciated by those skilled in the art that Fig.15 The data processing system 1500 shown is merely an example, and in other embodiments, the data processing system 1500 may have a different configuration (eg, include more components, have fewer components, etc.).

[0153] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for use by developers, or may be included in another software application, such as a terminal or computer operating system or a portable computing device operating system. In general, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, it is understood by the skilled person that the functionality of a software application may be distributed among multiple routines, objects, and / or components to achieve the same functionality required herein.

[0154] In addition, it is also understood that if the methods and systems of the present invention are implemented entirely or in part by a computing system, any suitable computing system architecture may be used. This includes stand-alone computers, network computers, dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include (but not limited to) any suitable arrangement of computers or information processing hardware capable of implementing the described functions.

[0155] Some embodiments of the present invention relate to performing inference using a trained neural network-based ranking model (trained using one or more method embodiments of the present invention). To this end, some embodiments of the present invention provide a computer-implemented method for operating a neural network-based ranking model (trained using one or more method embodiments of the present invention). The method includes: processing a query and a document data set (data related to a plurality of documents) using a neural network-based ranking model that has been trained to determine a result. The method may further include: presenting (e.g., displaying) the result. The result may include a ranked or ordered list of documents.

[0156] Those skilled in the art will appreciate that the embodiments described and / or illustrated in the present invention may be varied and / or modified to provide other embodiments of the present invention. Therefore, the embodiments described and / or illustrated in the present invention should be considered in all respects as illustrative and not restrictive. Examples of optional features of certain embodiments of the present invention are provided in the summary and description. Some embodiments of the present invention may include one or more of these optional features (some of which are not specifically described in the drawings). Some embodiments of the present invention may lack one or more of these optional features (some of which are not specifically described in the drawings). For example, the neural network-based sorting model of the present invention may have a different network architecture (i.e., different from the architecture specifically described or illustrated).

Claims

1. A computer-implemented method for training a neural network-based ranking model, include: Performing a training data augmentation operation on the training data set to generate a synthetic training data set; as well as Using the training data set and the synthetic training data set to train a neural network-based ranking model; Wherein, for each query in the plurality of queries, the training data set includes its corresponding query-document data and corresponding relevance judgment data; the query-document data of the query includes data associated with a plurality of query-document pairs of the query; the relevance judgment data of the query includes one or more sets of user feedback data associated with the query; wherein the training data set has an unbalanced training data distribution such that the amount of available relevance judgment data for at least some of the multiple queries is different, and wherein the synthetic training data set is suitable for reducing the imbalance in the training data distribution of the training data set.

2. The computer-implemented method of claim 1, wherein the unbalanced training data distribution as a whole follows a long-tail distribution or a heavy-tail distribution.

3. The computer-implemented method of claim 1 , wherein the training data augmentation operation include: For each query in the plurality of queries, determining a corresponding representation of the query; For each query in the plurality of queries, determining one or more corresponding neighboring queries based on the determined representations of the plurality of queries; as well as For one or more queries of the plurality of queries, synthetic training data is generated based on the relevance determination data associated with the query and the relevance determination data associated with one or more neighboring queries of the query.

4. The computer-implemented method according to claim 3, in: Each query in the plurality of queries corresponds to a plurality of query-document pairs having corresponding features; and Therein, the corresponding representation of the query is determined according to the corresponding feature of the query. 5 . The computer-implemented method of claim 4 , wherein said determining said corresponding representation of said query is based on statistics of said corresponding features of said query. The computer-implemented method of claim 5 , wherein the determining the corresponding representation of the query is based on an average of the corresponding features of the query. 7 . The computer-implemented method of claim 3 , wherein the determination of the one or more corresponding neighbor queries is based on a k-nearest neighbor approach.

8. The computer-implemented method of claim 3, wherein the synthetic training data is generated for a query include: (a) sampling a set of user feedback data associated with the query from multiple sets of user feedback data associated with the query to obtain a first data sample; (b) selecting one of the neighboring queries associated with the query, and sampling a set of user feedback data associated with the selected neighboring query from a plurality of sets of user feedback data associated with the selected neighboring query to obtain a second data sample; as well as (c) synthesizing a data sample based on the first data sample and the second data sample.

9. The computer-implemented method of claim 8, wherein the synthesis of the data samples is based on: in, l ′ is a synthetic data sample, is the first data sample, is the second data sample and λ is a hyperparameter.

10. The computer-implemented method of claim 9, wherein 0<λ<1.

11. The computer-implemented method of claim 8, wherein the synthetic training data is generated for one or more queries in the plurality of queries, respectively, further comprising: include: Repeat steps (a) to (c) to synthesize multiple data samples.

12. The computer-implemented method of claim 11, wherein the number of times steps (a) to (c) are repeated for each of the plurality of queries depends on the amount of user feedback data associated with the query.

13. The computer-implemented method of claim 11 , further comprising: include: determining, based on the relevance determination data of the plurality of queries, an occurrence frequency or a relative occurrence frequency of each of the plurality of queries; The number of times steps (a) to (c) are repeated for each of the plurality of queries depends on the determined occurrence frequency or the relative occurrence frequency of the corresponding query.

14. The computer-implemented method of claim 13, wherein the relative frequency of occurrence of a query is determined based on: in, The tail metric corresponding to the query, t i is the number of sets of user feedback data for the query.

15. The computer-implemented method of claim 14, wherein the number of times steps (a) to (c) are repeated for each of the plurality of queries is proportional to a weighting factor w i Associated, the weighting factor w i Defined as in, w e is a hyperparameter that controls the overall weight of synthetic data samples, T max and T min are the maximum and minimum values ​​of T, respectively, and w c is the threshold value.

16. The computer-implemented method of claim 1, wherein the neural network-based ranking model include: a first model branch having a first multilayer perceptron and a first predictor operably coupled to the first multilayer perceptron; a second model branch having a second multilayer perceptron and a second predictor operably coupled to the second multilayer perceptron; as well as A combiner is configured to combine an output of the first predictor and an output of the second predictor.

17. The computer-implemented method of claim 16, wherein the combiner is configured to apply a weighting to the output of the first predictor, and / or to apply a weighting to the output of the second predictor.

18. The computer-implemented method of claim 16, wherein the training of the neural network-based ranking model include: Using the neural network-based ranking model, a ranking or scoring operation is performed based on the training data set and the synthetic training data set.

19. The computer-implemented method of claim 18, wherein the ranking or scoring operation include: Processing the training data set using the first model branch; as well as The combination of the training dataset and the synthetic training dataset is processed using the second model branch.

20. The computer-implemented method of claim 19, further comprising: include: Determining a ranking or scoring loss based on the ranking or scoring operation performed 21. The computer-implemented method of claim 18, wherein the training of the neural network-based ranking model further include: Using the neural network-based ranking model, a contrastive learning operation is performed based on the training data set and / or the synthetic training data set.

22. The computer-implemented method of claim 21, wherein the contrastive learning operation include: For one or more of the plurality of queries, performing a data perturbation operation on data associated with each of the plurality of query-document pairs for the query to generate respective enhanced data for the data of each of the plurality of query-document pairs; as well as The augmented data is processed using the neural network-based ranking model.

23. The computer-implemented method of claim 22, wherein the data perturbation operation include: For each of the data for each of the plurality of query-document pairs, generating: a first set of enhanced data having a first noise injection level; as well as A second set of enhanced data having a second noise injection level that is different than the first noise injection level.

24. The computer-implemented method of claim 22, wherein the neural network-based ranking model further comprises: include: a first projector operatively coupled to the first multilayer perceptron; as well as a second projector operatively coupled to the second multilayer perceptron; as well as Wherein, the comparative learning operation further includes: processing augmented data using the first multilayer perceptron and the first projector; and The augmented data is processed using the second multilayer perceptron and the second projector.

25. The computer-implemented method of claim 22, further comprising determining a contrastive loss based on the performance of the contrastive learning operation 26. The computer-implemented method of claim 22, further comprising performing a joint optimization operation based on the performing of the ranking or scoring operation and the performing of the contrastive learning operation.

27. The computer-implemented method of claim 26, wherein the joint optimization operation comprises jointly optimizing a ranking or scoring loss associated with performing the ranking or scoring operation. and the contrastive loss associated with performing the contrastive learning operation 28. The computer-implemented method of claim 27, wherein the joint optimization comprises optimizing a total loss It is equal to Where γ is a hyperparameter.

29. A system for training a neural network-based ranking model, include: one or more processors; as well as A memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising instructions for executing or facilitating the execution of the computer-implemented method as recited in claim 1.

30. A computer-implemented method for operating a neural network-based ranking model trained using the method of claim 1, include: The query and document dataset are processed using the neural network-based ranking model to determine a result.

Images