Training target model

Abstract

The present disclosure provides a method and apparatus for training a target model. A first data set may be used to train at least one reference model. A second data set and a third data set may be scored using the at least one reference model. The scored second data set may be used to train the target model. The scored third data set may be used to optimize the target model.

Description

Technical Field

[0001] The present disclosure relates to methods, apparatus, and computer-readable media for training a target model. Background Art

[0002] With the development of technologies such as machine learning, deep learning, and neural networks, a variety of models based on these technologies have been continuously developed and applied. Taking search engines as an example, using search engines to find specific content on the Internet has become a part of computer users' daily lives. After receiving a user's search query, the search engine first uses a trained matching model to recall a specific number of documents related to the query from a pre-established index database. These documents are then processed, such as relevance filtering and sorting, and finally a series of documents with the highest ranking are selected to be presented to the user. Since all these subsequent processing are performed only on the recalled documents, the matching model needs to be trained to be able to recall the documents that are most relevant to the query. In addition, as an early step in the search process, the matching model needs to recall documents quickly enough to cope with the intensive user query requests and the user's demand for immediate responses. Summary of the Invention

[0003] This summary is provided to introduce a set of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] Embodiments of the present disclosure provide a method and apparatus for training a target model. A first data set may be used to train at least one reference model. A second data set and a third data set may be scored using the at least one reference model. The scored second data set may be used to train the target model. The scored third data set may be used to optimize the target model.

[0005] It should be noted that one or more of the above aspects include features described in detail below and particularly pointed out in the claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of the various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The disclosed aspects will be described below with reference to the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0007] Figure 1 is a schematic diagram of an exemplary fast matching model.

[0008] Figure 2 An exemplary process of training a target model by using a reference model according to an embodiment of the present disclosure is shown.

[0009] Figure 3 is a schematic diagram of an exemplary underlying cross-matching model.

[0010] Figure 4 An exemplary process of training a reference model through multi-task learning according to an embodiment of the present disclosure is shown.

[0011] Figure 5 An exemplary process of optimizing a target model according to an embodiment of the present disclosure is shown.

[0012] Figure 6 is a flowchart of an exemplary method for training a target model according to an embodiment of the present disclosure.

[0013] Figure 7 An exemplary apparatus for training a target model according to an embodiment of the present disclosure is shown.

[0014] Figure 8 An exemplary apparatus for training a target model according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0015] The present disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that the discussion of these embodiments is only for enabling those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0016] Currently, search engines often use fast matching models to retrieve documents relevant to a query from a pre-built index database. In this article, a fast matching model refers to a lightweight, bottom-level separable model that can independently transform queries and documents into representation vectors in a common inner product space without requiring the underlying model to know each other. Common fast matching models include, for example, the Deep Structured Semantic Model (DSSM) and the Convolutional Deep Structured Semantic Model (CDSSM).

[0017] Figure 1 is a schematic diagram of an exemplary fast matching model 100. Figure 1As shown, the input of the fast matching model 100 may include a query 110 and a document 120. In one aspect, for query 110, the fast matching model 100 may include: an embedding layer 112 for converting each word in the input word sequence into a feature vector; a convolutional layer 114 for extracting sliding-window-based contextual features around each word; a pooling layer 116 for selecting the most important contextual features; and a semantic layer 118 for representing a high-level semantic feature vector of the input word sequence. In another aspect, for document 120, feature extraction may first be performed. The extracted features may include, for example, at least one of the following: keywords 122 representing the core topic of document 120; a document title 124 indicating the title of document 120; a uniform resource locator (URL) 126 indicating the internet address of document 120; a description 128 summarizing the main content of document 120; and a landing page (LP) title 130 indicating the title of the LP corresponding to document 120. In this context, an LP refers to the page corresponding to a link that a user reaches after clicking on the link on the search results page. For the above features extracted from the document 120, the fast matching model 100 may include an embedding layer 132, a convolution layer 134, a pooling layer 136, and a semantic layer 138, the functions of which are similar to those of the corresponding layers for the query 110. In addition, the fast matching model 100 may also include a scoring layer 180, which is used to determine the correlation between the feature vector of the query 110 output from the semantic layer 118 and the feature vector of the document 120 output from the semantic layer 138. It should be understood that Figure 1 The fast matching model 100 shown is only an example of an existing fast matching model. According to actual application requirements, the fast matching model can have any other structure and can include more or fewer layers.

[0018] In some cases, the fast matching model can pre-store feature vectors calculated for a large number of documents in the index database to reduce the amount of online calculation. Figure 1 The embedding layer 132, convolution layer 134, pooling layer 136 and semantic layer 138 shown in FIG. 1 respectively calculate the feature vectors of these documents and store them in the index database. When a query request from a user is received, the fast matching model can only perform feature vector calculation on the query, for example, by Figure 1The embedding layer 112, convolutional layer 114, pooling layer 116, and semantic layer 118 are shown. Then, by performing relevance matching between the query's feature vector and the feature vectors of documents stored in the index database, documents matching the query are retrieved. For example, the relevance matching can be efficiently performed using a nearest neighbor search algorithm. Thus, through the fast matching model, documents relevant to the input query can be quickly retrieved from the index database.

[0019] like Figure 1 As shown, the fast matching model 100 separates the query side and the document side, and performs vector conversion on the query 110 and the document 120 respectively. Since the query and the document do not interact until the scoring layer, the fast matching model often loses important information for the matching between the query and the document when performing the vector conversion, which will greatly limit the performance of the matching model and thus affect the accuracy of the recalled documents. In addition, the fast matching model is usually trained on training data with manually provided relevance annotations. Such training data is expensive and time-consuming to collect, and therefore the amount is limited. However, the performance of the fast matching model depends on whether there is a large amount of training data with relevance annotations.

[0020] Therefore, it is desirable to improve the performance of fast matching models to more accurately and quickly recall documents relevant to the query. However, there are many challenges to such improvement.

[0021] In one aspect, one can attempt to use a matching model with a more complex structure instead of a fast matching model to achieve better matching accuracy. Such a matching model can be, for example, a bottom-level cross-matching model. In this article, a bottom-level cross-matching model refers to a model in which the input query and document interact immediately after the embedding layer. Common bottom-level cross-matching models include the Deep Crossing Model, the Decision Tree Ensemble Model, the Deep Cross Network Model, etc. Since in the bottom-level cross-matching model, the query and document interact immediately after the embedding layer, this structure of the bottom-level cross-matching model can provide better performance than the fast matching model, but at the same time, the computational complexity is greatly increased. In practical applications, query requests from users are extremely intensive. As a preliminary step in the search process, recalling documents that match the query from the index database must be performed quickly. Therefore, the bottom-level cross-matching model cannot be directly deployed to perform document recall.

[0022] On the other hand, some technologies have proposed using data without relevance annotations to train fast matching models as an alternative to manually annotated training data. For example, some have proposed using search log data to train fast matching models. In this case, user clicks are often used as a substitute for manually provided relevance annotations. For example, query-document pairs clicked by users are considered relevant, while query-document pairs synthesized from a query and other randomly selected documents are considered irrelevant. However, this approach of using search log data presents many problems. For example, the arbitrariness and subjectivity of user behavior can lead to a deviation between user clicks and actual relevance, and synthesized irrelevant query-document pairs are likely to contain actually relevant query-document pairs, both of which reduce the accuracy of the training data and cause "contamination" of the training data. Furthermore, to distinguish relevance at a finer granularity, the manually provided annotations are often hierarchical, for example, using six values ​​from "0" to "5" to represent different levels of relevance, where higher values ​​indicate greater relevance. Such annotations may be difficult to approximate by processing search log data based on user clicks.

[0023] The embodiments of the present disclosure propose to improve the performance of the target model through an improved training process. For example, the target model can be trained by using a reference model. In this article, the target model refers to a model that is simple in structure and can be deployed, such as a fast matching model, and the reference model refers to a model that can be used to assist in training the target model and has a relatively complex structure and cannot usually be directly deployed, such as an underlying cross-matching model. It should be understood that although the following discussion involves an example of using an underlying cross-matching model to train a fast matching model, the embodiments of the present disclosure are not limited to this, but other types of reference models can be used in a similar manner to train other types of target models.

[0024] In one aspect, according to an embodiment of the present disclosure, a reference model can be used to score a large number of unlabeled data sets to obtain a large amount of training data for training a target model. For example, an unlabeled data set may include search log data, which may include queries and documents from a large number of search processes of a search engine, and thus its quantity is huge. Since the reference model may be a model with higher performance, its scoring of samples in the data set will have higher accuracy and can also better approximate human annotations. The large amount of training data obtained can be used to train the target model. Since the amount of these training data will greatly exceed the available human-labeled training data, and the scores in these training data have higher accuracy, this will help to train a target model with better performance.

[0025] In another aspect, embodiments of the present disclosure can further optimize a trained target model using a reference model. Another set of labeled data can be scored using the reference model to obtain a scored labeled data set. Each sample in this scored labeled data set includes both a label and a score provided by the reference model. These samples, both labeled and scored, can be used to optimize the trained target model.

[0026] In another aspect, embodiments of the present disclosure also provide an effective training method for the reference model. For example, the reference model can be trained by jointly learning multiple related tasks to enable it to distinguish relevance with higher accuracy and finer granularity.

[0027] Figure 2 An exemplary process 200 of training a target model by using a reference model according to an embodiment of the present disclosure is shown. As an example, the target model may be a fast matching model.

[0028] First, a first dataset 210 for training a reference model may be obtained. First dataset 210 may be, for example, a labeled dataset. First dataset 210 may include multiple samples. Each sample may include a query, a document, and a label, for example, represented as a triple <query, document, label>, where the label may indicate the relevance between the query and the document.

[0029] The annotations in the first data set 210 may be added manually or in any other way. The relevance of each query-document pair in the first data set 210 may be scored, and an annotation indicating the relevance of the query-document pair may be given. The annotations added manually are relatively credible, so they are considered "strong annotations". In addition, in order to distinguish relevance at a fine-grained level, the annotations in the first data set 210 are usually hierarchical enumeration-type annotations, for example, different levels of relevance are represented by a set of relevance values, where a larger relevance value indicates a greater relevance. As an example, the relevance value of the annotation may be {0, 1, 2, 3, 4, 5}, where "0" indicates no relevance and "5" indicates the most relevance.

[0030] In one case, for the same query-document pair, there may be two types of annotations. The first type of annotation is the document copy annotation, which indicates the relevance between the query and the document copy. In this article, the document copy refers to the information about the document that the user can see on the search results page. The second type of annotation is the landing page annotation, which indicates the relevance between the query and the landing page. In this article, the landing page refers to the page that the user arrives at after clicking the link corresponding to a document on the search results page. The "annotation" in the sample triple <query, document, annotation> may include the document copy annotation and the landing page annotation, respectively, or may be a comprehensive annotation obtained based on the document copy annotation and the landing page annotation.

[0031] At 220, a set of tasks can be constructed to convert each enumerated label in the first dataset 210 into a set of binary labels to obtain a converted first dataset 210. This conversion can more fully utilize the fine-grained information provided by the enumerated labels. In this article, the binary labels can include positive labels, such as "1", indicating that the query and document are relevant, and negative labels, such as "0", indicating that the query and document are irrelevant. When training a matching model, converting the enumerated labels into binary labels can improve the performance of the matching model.

[0032] Typically, an enumerated label is uniquely converted into positive and negative labels. For example, in the case where a label has 6 correlation values ​​from "0" to "5", the label with a correlation value of "0" is converted into a negative label, and the labels with a correlation value greater than "0" are all converted into positive labels. However, this conversion does not take into account the discrimination between labels with values ​​greater than "0". For example, labels with correlation values ​​of "2" and "3" are both converted into positive labels. In contrast, in an embodiment of the present disclosure, an enumerated label is not uniquely converted into positive and negative labels, but is converted into a set of binary labels through a set of tasks to increase the discrimination between different correlation values.

[0033] At 230, at least one reference model may be trained using the transformed first dataset 210. It should be noted that since the operation of constructing the task at 220 is optional, the first dataset 210 may also be directly used to train at least one reference model.

[0034] In one embodiment, the reference model may be, for example, an underlying cross-matching model. Figure 3 is a schematic diagram of an exemplary underlying cross-matching model 300. Figure 3As shown, the inputs of the underlying cross-matching model 300 may include a query 310 and a document 320. In one aspect, the underlying cross-matching model 300 may include an embedding layer 340 for transforming the query 310. In another aspect, feature extraction may first be performed on the document 320. The extracted features may include, for example, at least one of the following: keywords 322, document title 324, URL 326, description 328, and document title 330. For the features extracted from the document 320, the underlying cross-matching model 300 may include embedding layers 342, 344, 346, 348, and 350 for transforming each feature of the document 320. The outputs of the embedding layers 340 to 350 may then be provided to a stacking layer 360 for stacking into a feature vector and providing it to a residual layer 370. The residual layer 370 is composed of residual units, which can transform the original input features through, for example, two layers of rectified linear units (ReLUs) and then add the transformed input features to the original input features by dimension. Finally, the feature vector is scored by the scoring layer 380 to indicate the relevance between the query 310 and the document 320. It should be understood that Figure 3 The bottom cross-matching model 300 shown is only an example of a bottom cross-matching model. According to actual application requirements, the bottom cross-matching model can have any other structure and can include more or fewer layers.

[0035] It should be understood that the embodiments of the present disclosure may use a single reference model or multiple reference models to train the target model. Therefore, a single reference model may be trained at 230 or multiple reference models may be trained separately. In the case of multiple reference models, these reference models may have the same model structure, for example, all are deep cross models, or these reference models may have different model structures, for example, a combination of deep cross models, decision tree ensemble models, etc. In the case where multiple reference models have different model structures, since each reference model has its own advantages, the greater the difference in model structure, the stronger the performance of the model ensemble obtained by the subsequent combination.

[0036] In process 200, after at least one reference model is trained at 230, a second dataset 240 may be scored using the at least one reference model. This second dataset 240 will be used to form training data for training the target model. Second dataset 240 may be, for example, an unlabeled dataset. Second dataset 240 may include multiple samples, each of which includes at least a query and a document and has a structure such as <query, document>. The samples in second dataset 240 may be based on, for example, search log data.

[0037] At least one reference model can score each sample in the second data set 240 to obtain a relevance score for the sample. Here, the relevance score obtained by the reference model is also referred to as a target score, which indicates the relevance between the query and the document in the sample and serves as a reference for subsequent training of the target model. The scored second data set 240 forms a first scoring data set 250. The samples in the first scoring data set 250 can have a structure such as <query, document, target score>. Assume that the target score of the i-th sample in the first scoring data set 250 is represented as s i , where 0≤s i ≤1, then s i The larger the value, the more relevant the query is to the document. Since the target score is given by the reference model, it is slightly less reliable than the human-provided annotations, so it is also called "weak annotation".

[0038] In one embodiment, if the at least one reference model includes more than one reference model, then for each sample in the second data set 240, the relevance between the query and the document in the sample can be scored using the at least one reference model to obtain at least one initial score for the sample. Subsequently, a target score for the sample can be generated based on the at least one initial score. For example, if the at least one reference model includes two reference models, the sample can be scored using each reference model to obtain two initial scores for the sample. Then, a target score for the sample can be generated based on the two initial scores. In one example, the two initial scores can be arithmetic averaged, and the result obtained is used as the target score for the sample.

[0039] As previously mentioned, the samples in the second dataset 240 can be based on search log data. Since the amount of search log data is enormous, by scoring it with the reference model, a large amount of scored search log data can be obtained. Consequently, the first scored dataset 250 will include a large amount of training data that can be used to train the target model.

[0040] At 260, the target model can be trained using the first scoring dataset 250. The target model can be, for example, a fast matching model. In one embodiment, for each sample in the first scoring dataset 250, the target model can be used to score the sample to obtain a relevance score for the sample. Here, the relevance score obtained by the target model can also be referred to as a prediction score. The prediction loss of the sample can then be calculated using both the target score provided by the reference model and the prediction score provided by the target model, and the target model can be trained by minimizing the prediction loss.

[0041] During process 200, a third dataset 270 may also be scored using at least one reference model. This third dataset 270 will be used to form optimized training data for optimizing the trained target model. Third dataset 270 may be, for example, a labeled dataset. Third dataset 270 may include multiple samples, each of which includes at least a query, a document, and a label provided manually or otherwise, and has a structure such as <query, document, label>, where the label indicates the relevance between the query and the document. Each sample in third dataset 270 may be scored using at least one reference model to obtain a relevance score for the sample. The relevance score obtained using the reference model is also referred to as a target score. It indicates the relevance between the query and the document in the sample and serves as a reference for subsequent optimization of the target model. The scored third dataset 270 forms a second scored dataset 280. Samples in second scored dataset 280 may have a structure such as <query, document, label, target score>. Scoring third dataset 270 using at least one reference model may be similar to scoring second dataset 240 using at least one reference model.

[0042] At 290, the target model trained at 260 can be optimized using the second scoring dataset 280. For each sample in the second scoring dataset 280, the sample can be scored by the target model to obtain a relevance score for the sample, which can also be referred to as a prediction score. The prediction loss corresponding to the sample can then be calculated using a combination of the annotations in the sample, the target score provided by the reference model, and the prediction score provided by the target model, and the target model can be optimized by minimizing the prediction loss.

[0043] The target model ultimately obtained through process 200 can be deployed online for performing document recall, while the at least one reference model is only run offline for training the target model. It should be understood that, in this context, using the reference model to train the target model can include both the initial training operation of the target model at 260 and the optimization operation of the trained target model at 290.

[0044] Embodiments of the present disclosure propose a method for training a reference model using multi-task learning (MTL). In this context, MTL refers to using enumerated annotations to construct multiple related tasks and training the model by jointly learning these tasks. Using MTL during reference model training can more fully utilize the fine-grained information provided by enumerated annotations.

[0045] Figure 4An exemplary process 400 for training a reference model by MTL according to an embodiment of the present disclosure is shown. The process 400 may correspond to Figure 2 Operations 220 and 230 in FIG.

[0046] At 402, a set of tasks may be used to convert the enumerated labels in the samples of the data set used to train the reference model into a set of binary labels. Figure 2 The first data set 210 in the set of binary labels. The number of the set of binary labels can be equal to or less than the dimension of the value of the enumerated label. The enumerated label can include multiple relevance values, such as {0,1,2,3,4,5}. The binary label can include a positive label indicating that the query is relevant to the document, for example, "1", and a negative label indicating that the query is not relevant to the document, for example, "0". A set of tasks for converting an enumerated label into a set of binary labels can convert the enumerated label into positive labels and negative labels based on respective cutoff values. In this example, the cutoff value for each task can be taken from, for example, one of {0,1,2,3,4}. In a particular task, enumerated labels with relevance values ​​less than or equal to the cutoff value are converted into negative labels, while enumerated labels with relevance values ​​greater than the cutoff value are converted into positive labels.

[0047] In one embodiment, the set of tasks may include a main task and at least one auxiliary task. The main task may refer to a task whose cutoff value is a critical value that divides the relevance between the query and the document into relevant and irrelevant values ​​among a plurality of relevance values ​​of an enumerated annotation, and the auxiliary task may refer to a task whose cutoff value is a value other than the critical value among the plurality of relevance values. A relevance value less than or equal to the critical value among the plurality of relevance values ​​may indicate that the query is irrelevant to the document, while a relevance value greater than the critical value among the plurality of relevance values ​​may indicate that the query is relevant to the document. For example, for document copy annotation, the critical value may be "0", so that a relevance value of "0" indicates that the query is irrelevant to the document, while a relevance value of "1" or greater indicates that the query is relevant to the document. In addition, for example, for landing page annotation, the critical value may be "1", so that relevance values ​​of "0" and "1" indicate that the query is irrelevant to the document, while a relevance value of "2" or greater indicates that the query is relevant to the document. Table 1 shows an exemplary annotation division based on main tasks and auxiliary tasks 1-4. In this example, the relevance values ​​of the enumerated annotation are {0, 1, 2, 3, 4, 5} and the critical value is "0". The cutoff value of the main task is "0" and can distinguish between annotations with a correlation value of "0" and annotations with a correlation value greater than "0", while the cutoff values ​​of auxiliary tasks 1-4 are "1", "2", "3" and "4" respectively and can further distinguish annotations with a correlation value greater than "0".

[0048]

[0049]

[0050] Table 1

[0051] Table 1 shows which enumerated labels are converted to negative labels and which are converted to positive labels by each task. For example, the cutoff value of auxiliary task 3 is "3". By auxiliary task 3, enumerated labels with correlation values ​​of {0, 1, 2, 3} are converted to negative labels "0", and enumerated labels with correlation values ​​of {4, 5} are converted to positive labels "1".

[0052] In another example, the critical value may be "1". In this case, the cutoff value for the primary task is "1". In the primary task, an enumerated label with a correlation value of {0,1} is converted into a negative label "0", and an enumerated label with a correlation value of {2,3,4,5} is converted into a positive label "1". The cutoff values ​​for auxiliary tasks 1-4 are "0", "2", "3" and "4", respectively. For example, the cutoff value for auxiliary task 1 may be "0". In auxiliary task 1, an enumerated label with a correlation value of {0} is converted into a negative label "0", and an enumerated label with a correlation value of {1,2,3,4,5} is converted into a positive label "1".

[0053] Through the above-mentioned set of tasks including the main task and the auxiliary task, an enumeration type label can be converted into a set of binary labels. In addition, the set of tasks can distinguish enumeration type labels with correlation values ​​greater than "0", thereby making use of fine-grained hierarchical labels. Taking the sample <query m, document k, 2> as an example, "2" is an enumeration type label indicating the correlation between "query m" and "document k". Through the above-mentioned set of tasks as shown in Table 1, the enumeration type label "2" can be converted into a set of binary labels "1", "1", "0", "0" and "0" corresponding to the main task and auxiliary tasks 1-4 respectively. Taking the sample <query m, document k, 3> as an example, "3" is an enumeration type label indicating the correlation between "query m" and "document k". Through the above-mentioned set of tasks as shown in Table 1, the enumeration type label "3" can be converted into another set of binary labels "1", "1", "1", "0" and "0" corresponding to the main task and auxiliary tasks 1-4 respectively. It can be seen that, through the above set of tasks, the enumeration type annotation "2" and the enumeration type annotation "3" can be converted into two different sets of binary annotations.

[0054] At 404, after the enumerated annotations in a sample are converted into a set of binary annotations by a set of tasks, a set of derived samples can be created by combining the query and document in the sample with the set of binary annotations. In this document, a derived sample refers to a sample that includes at least a query, a document, and a binary annotation, wherein the binary annotation is converted from the enumerated annotation by the constructed tasks.

[0055] Continuing with the example <query m, document k, 2>, through the primary and auxiliary tasks described in Table 1, the enumerated label "2" can be converted into a set of binary labels: "1," "1," "0," "0," and "0." By combining "query m" and "document k" with this set of binary labels, a set of derived samples can be created, such as <query m, document k, 1>, <query m, document k, 1>, <query m, document k, 0>, <query m, document k, 0>, and <query m, document k, 0>.

[0056] After creating a set of derived samples, process 400 can proceed to train a reference model using the set of derived samples.

[0057] At 406, the reference model may be used to score the derived samples to obtain a set of prediction scores corresponding to the derived samples. Here, the prediction score refers to a score provided by the reference model after scoring the relevance between the query and the document of each derived sample.

[0058] Continuing with the previous example, the reference model can score a set of derived samples <query m, document k, 1>, <query m, document k, 1>, <query m, document k, 0>, <query m, document k, 0>, and <query m, document k, 0> of the sample <query m, document k, 2> to obtain a corresponding set of prediction scores, for example, denoted as s0, s1, s2, s3, and s4, respectively.

[0059] At 408 , a set of prediction losses respectively corresponding to the set of derived samples may be calculated based on the set of binary labels and the set of prediction scores. It should be understood that embodiments of the present disclosure are not limited to any particular manner of calculating prediction losses.

[0060] Continuing with the previous example, a set of prediction losses l0, l1, l2, l3, and l4 corresponding to the set of derived samples <query m, document k, 1>, <query m, document k, 1>, <query m, document k, 0>, <query m, document k, 0>, and <query m, document k, 0>, respectively, can be calculated based on the set of binary labels "1", "1", "0", "0", and "0" and the set of prediction scores s0, s1, s2, s3, and s4. For example, taking the second derived sample <query m, document k, 1> in the set of derived samples as an example, the prediction loss l1 of the derived sample can be calculated using the binary label "1" of the derived sample and the prediction score s1 of the derived sample.

[0061] At 410 , a composite predicted loss may be generated based on the set of predicted losses.

[0062] In one embodiment, each prediction loss in the set of prediction losses may be directly summed to generate a composite prediction loss.

[0063] In another embodiment, a weighting coefficient for each prediction loss in the set of prediction losses may be set first, and then the set of prediction losses may be weighted and summed based on the set weighting coefficients to generate a comprehensive prediction loss. For example, the weighting coefficients may be set based on the tasks to which the derived samples correspond. For example, for the primary task, the weighting coefficient may be set to 0.5, while for the auxiliary tasks, the weighting coefficients may be set equally, for example, to (1-0.5) / n, where n is the number of auxiliary tasks.

[0064] At 412 , the reference model may be optimized by minimizing the composite prediction loss.

[0065] It should be understood that when more than one reference model is used in the embodiments of the present disclosure, different reference models can be trained separately through the above process 400.

[0066] According to an embodiment of the present disclosure, after the reference model is trained, it is possible to train the unlabeled data set, such as Figure 2 The second data set 240 in the dataset is scored to obtain a scored unlabeled data set for training the target model, for example Figure 2The first scoring data set 250 in the dataset for training the target model may include multiple samples of the structure of <query, document, target score>, where the target score is provided by the reference model after scoring the relevance between the query and the document in the sample. In one embodiment, when training the target model, in order to effectively utilize the target scores of each sample provided by the reference model, the target scores may be first converted to obtain derived scores. In this article, the derived score refers to the score that indicates the relevance between the query and the document in each sample and is directly used to train the target model. In the following discussion, the target score of the i-th sample in the dataset for training the target model is denoted as s i , and the derived score of this sample is denoted as y i .

[0067] In one embodiment, the score y is derived i It can be the target score s i The original value of is shown in the following formula (1):

[0068] y i =s i (1)

[0069] In another embodiment, the target score s can be calculated based on the threshold t1. i Convert to obtain the binary derivative y i "1" and "0", as shown in the following formula (2):

[0070]

[0071] When training the target model, the target model can be used to score the relevance between the query and the document in each sample of the data set used to train the target model to obtain the prediction score of each sample. The prediction score of the i-th sample can be expressed as In one embodiment, the loss of the i-th sample l i It is calculated as the weighted square loss as shown in the following formula (3):

[0072]

[0073] Among them, w i is the weight corresponding to the i-th sample set when calculating the loss of the target model, where 0≤w i ≤1. The weight w can be set, for example, according to one of the following formulas (4)-(6): i :

[0074]

[0075] w i=|2s i -1| p (5)

[0076] w i =1 (6)

[0077] Among them, t2, t3 and p are system settings used to calculate the weight w i Parameters.

[0078] In one embodiment, a data set including multiple samples based on search log data can be scored by at least one reference model, and then the scored data set can be used to train a target model. Since the amount of search log data is huge, the scored data set can provide a large amount of training data for training the target model. Although the search log data does not have human-provided annotations, after being scored by the reference model, each sample can have a target score indicating the relevance between the query and the document, and thus these target scores can be used to effectively train the target model. By scoring the search log data and using the scored search log data to train the target model, the reliance on human-annotated training data can be reduced. In addition, compared with the method of using user clicks as an alternative to relevance annotations, this method of scoring the search log data by the reference model can be more accurate.

[0079] According to an embodiment of the present disclosure, after the target model is initially trained, another dataset scored by the reference model can be used to optimize the target model. For example, the reference model can be used to optimize the labeled dataset, such as Figure 2 The third data set 270 in the example is scored to obtain a scored labeled data set for optimizing the target model, for example Figure 2 The second scoring dataset 280 in the target model. The dataset used to optimize the target model may include multiple samples with a structure of <query, document, label, target score>, wherein the label may be a relevance value indicating the relevance between the query and the document of the sample provided in advance manually or in other ways, and the target score is provided by the reference model after scoring the relevance between the query and the document of the sample.

[0080] Figure 5 An exemplary process 500 for optimizing a target model according to an embodiment of the present disclosure is shown. The process 500 may correspond to Figure 2 Operation 290 in.

[0081] At 502, the relevance between the query and the document in each sample of the data set used to optimize the target model can be scored by the target model to obtain a prediction score for the sample. The prediction score of the i-th sample can be expressed as

[0082] Process 500 can then calculate the prediction loss corresponding to the sample based on the combination of the sample's label and target score and the prediction score. The prediction loss of the i-th sample can be expressed as

[0083] In one embodiment for calculating the prediction loss, at 504 , the credibility of the sample may be determined based on whether a combination of the label and target score of the sample and the prediction score satisfies a predetermined rule.

[0084] The predetermined rule may at least be based on the annotations in the sample. The predetermined rule may include: when the annotations in the sample indicate that the query and document are related, the predicted score is greater than the target score. For example, the predetermined rule states that when the annotations in the sample indicate that the query and document are related, the predicted score obtained by the target model from scoring the relevance between the query and document in the sample should be as large as possible. Preferably, the predicted score should be greater than the target score provided by the reference model. The predetermined rule may also include: when the annotations in the sample indicate that the query and document are unrelated, the predicted score is less than the target score. For example, the predetermined rule states that when the annotations in the sample indicate that the query and document are unrelated, the predicted score obtained by the target model from scoring the relevance between the query and document in the sample should be as small as possible. Preferably, the predicted score should be less than the target score provided by the reference model. When the combination of the annotations, target score, and predicted score of the sample meets the predetermined rule, the sample is determined to be trustworthy. Otherwise, the sample is determined to be untrustworthy.

[0085] In one embodiment, when determining whether the combination of the sample's label, target score, and prediction score satisfies a predetermined rule, the label in the sample can be converted into a binary label. The binary label of the i-th sample can be represented as For example, the labels in the samples can be converted into binary labels through any of the main tasks and auxiliary tasks mentioned above.

[0086] In one embodiment, in order to effectively utilize the target scores of each sample provided by the reference model during the optimization of the target model, the target scores can be transformed to obtain derived scores. The derived score of the i-th sample can be expressed as y i The target scores in each sample can be converted in a similar manner to the training target model, for example, by converting the target scores into derived scores according to the above formula (1) or (2).

[0087] Then, at 506, a weight corresponding to the sample may be set based on the credibility of the sample. The weight corresponding to the i-th sample may be expressed as In one embodiment, the weight is set based on a predetermined criterion, which may include: the weight corresponding to a sample indicated as credible by the credibility is less than or equal to the weight corresponding to a sample indicated as uncredible by the credibility. In one embodiment, for a credible sample i, For untrustworthy sample i,

[0088] To facilitate the description of weights The embodiment of the present disclosure defines a sign function as shown in the following formula (7):

[0089]

[0090] Among them, 0≤θ≤1 is the hyperparameter set by the system.

[0091] According to an embodiment of the present disclosure, the weight It can be defined as the following formula (8):

[0092]

[0093] It should be understood that formulas (7) and (8) only describe the weights Other forms can also be used to describe the weights according to the embodiments of the present disclosure.

[0094] It can be seen that different from the weight w based on which the target model is trained i , which is only related to the target score and is the weight based on which the target model is optimized It is still related to the label. Therefore, the weights based on which the target model is optimized It can also be called annotation-aware weight.

[0095] At 508, the weights may be used to determine the To calculate the prediction loss. As mentioned above, the prediction loss of the i-th sample can be expressed as In one embodiment, the prediction loss It is defined as the weighted square loss, as shown in the following formula (9):

[0096]

[0097] At 510, the prediction loss Minimize to optimize the target model.

[0098] pass Figure 5In the process 500, the target model can be optimized by at least utilizing both the target score provided by the reference model and the correlation annotations included in the data set, thereby further improving the performance of the target model. The method for optimizing the target model according to an embodiment of the present disclosure aims to calculate the corresponding prediction loss based on both the target score provided by the reference model and the correlation annotations included in the data set, and optimize the target model by minimizing the prediction loss. When calculating the prediction loss, the correlation annotations can be used as a reference as described above. In the above steps 504 to 508, only one embodiment for calculating the prediction loss is exemplified. Under the idea of ​​calculating the prediction loss corresponding to the sample based on the combination of the sample's annotations, target scores, and prediction scores, the embodiments of the present disclosure can also cover any other embodiments for calculating the prediction loss based on this idea.

[0099] Figure 6 is a flowchart of an exemplary method 600 for training a target model according to an embodiment of the present disclosure.

[0100] At step 610 , at least one reference model may be trained using a first data set.

[0101] At step 620 , the second data set and the third data set may be scored respectively by at least one reference model.

[0102] At step 630 , the target model may be trained using the scored second dataset.

[0103] At step 640 , the target model may be optimized using the scored third data set.

[0104] In one embodiment, the first data set includes multiple samples, each sample includes at least a query, a document, and an enumerated label indicating the correlation between the query and the document, and the training of the at least one reference model includes, for each sample: converting the enumerated label in the sample into a set of binary labels through a set of tasks; creating a set of derived samples by combining the query and document in the sample and the set of binary labels; and using the set of derived samples to train the at least one reference model.

[0105] In one embodiment, the set of binary labels includes positive labels indicating that the query is relevant to the document and negative labels indicating that the query is irrelevant to the document, and the set of tasks converts the enumerated labels into positive labels or negative labels based on respective cutoff values.

[0106] In one embodiment, the value of the enumerated label is selected from multiple relevance values, and the set of tasks includes a main task and at least one auxiliary task, the cutoff value of the main task is a critical value among the multiple relevance values ​​that divides the relevance between the query and the document into relevant and irrelevant, and the cutoff value of the at least one auxiliary task is other values ​​among the multiple relevance values ​​except the critical value.

[0107] In one embodiment, the use of the set of derived samples to train the at least one reference model includes: scoring the set of derived samples respectively through the at least one reference model to obtain a set of prediction scores corresponding to the set of derived samples respectively; calculating a set of prediction losses corresponding to the set of derived samples respectively based on the set of binary labels and the set of prediction scores; generating a comprehensive prediction loss based on the set of prediction losses; and optimizing the at least one reference model by minimizing the comprehensive prediction loss.

[0108] In one embodiment, the second data set and the third data set respectively include multiple samples, each sample includes at least a query and a document, and the scoring includes, for each sample: scoring the relevance between the query and the document in the sample using the at least one reference model to obtain at least one initial score for the sample; and generating a target score for the sample based on the at least one initial score.

[0109] In one embodiment, the scored third data set includes multiple samples, each sample includes a query, a document, an annotation, and a target score, and the optimization includes, for each sample: scoring the relevance between the query and the document in the sample by the target model to obtain a prediction score for the sample; calculating a prediction loss corresponding to the sample based on a combination of the annotation and target score in the sample and the prediction score; and optimizing the target model by minimizing the prediction loss.

[0110] In one embodiment, the calculating the prediction loss includes: determining the credibility of the sample based on whether the combination satisfies a predetermined rule; setting a weight corresponding to the sample based on the credibility of the sample; and calculating the prediction loss based on the weight.

[0111] In one embodiment, the predetermined rule is at least based on the annotation.

[0112] In one embodiment, the predetermined rule includes: when the annotation indicates that the query and the document are relevant, the predicted score is greater than the target score; and when the annotation indicates that the query and the document are irrelevant, the predicted score is less than the target score.

[0113] In one embodiment, the weight is set based on a predetermined criterion, wherein the predetermined criterion includes: a weight corresponding to a sample indicated as credible by the credibility is less than or equal to a weight corresponding to a sample indicated as uncredible by the credibility.

[0114] In one embodiment, the target model is a fast matching model and the at least one reference model is an underlying cross matching model.

[0115] In one embodiment, the at least one reference model has the same model structure or has a different model structure.

[0116] In one embodiment, the second data set includes a plurality of samples based on search log data.

[0117] It should be understood that method 600 may also include any steps / processing for training the target model according to the above-mentioned embodiments of the present disclosure.

[0118] Figure 7 An exemplary apparatus 700 for training a target model according to an embodiment of the present disclosure is shown.

[0119] The device 700 may include: a reference model training module 710, used to train at least one reference model using a first data set; a scoring module 720, used to score the second data set and the third data set respectively using the at least one reference model; a target model training module 730, used to train the target model using the scored second data set; and an optimization module 740, used to optimize the target model using the scored third data set.

[0120] In one embodiment, the first data set includes multiple samples, each sample includes at least a query, a document, and an enumerated label indicating the correlation between the query and the document, and the reference model training module 710 is further configured to, for each sample: convert the enumerated label in the sample into a set of binary labels through a set of tasks; create a set of derived samples by combining the query and document in the sample and the set of binary labels; and use the set of derived samples to train the at least one reference model.

[0121] In one embodiment, the second data set and the third data set respectively include multiple samples, each sample includes at least a query and a document, and the scoring module 720 is further configured to, for each sample: score the relevance between the query and the document in the sample using the at least one reference model to obtain at least one initial score for the sample; and generate a target score for the sample based on the at least one initial score.

[0122] In one embodiment, the scored third data set includes multiple samples, each sample includes a query, a document, an annotation, and a target score, and the optimization module 740 is further configured to, for each sample: score the relevance between the query and the document in the sample by the target model to obtain a prediction score for the sample; calculate the prediction loss corresponding to the sample based on a combination of the annotation and target score in the sample and the prediction score; and optimize the target model by minimizing the prediction loss.

[0123] In one embodiment, the calculating the prediction loss includes: determining the credibility of the sample based on whether the combination satisfies a predetermined rule; setting a weight corresponding to the sample based on the credibility of the sample; and calculating the prediction loss based on the weight.

[0124] In addition, the apparatus 700 may further include any other modules configured for training a target model according to the above-mentioned embodiments of the present disclosure.

[0125] Figure 8 An exemplary apparatus 800 for training a target model according to an embodiment of the present disclosure is shown.

[0126] The apparatus 800 may include at least one processor 810. The apparatus 800 may further include a memory 820 connected to the processor 810. The memory 820 may store computer-executable instructions that, when executed, cause the processor 810 to perform any operation of the method for training a target model according to the above-described embodiments of the present disclosure.

[0127] The embodiments of the present disclosure may be implemented in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any operation of the method for training a target model according to the above-described embodiments of the present disclosure.

[0128] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operation in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0129] It should also be understood that all modules in the above-described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together in function.

[0130] Processor has been described in conjunction with various devices and methods.These processors can be implemented using electronic hardware, computer software or its arbitrary combination.Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraint imposed on the system.As an example, the processor provided in this disclosure, any part of the processor or any combination of processors can be implemented as microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), programmable logic device (PLD), state machine, gate logic, discrete hardware circuit and other suitable processing components configured for performing the various functions described in this disclosure.The function of the processor provided in this disclosure, any part of the processor or any combination of processors can be implemented as software performed by microprocessor, microcontroller, DSP or other suitable platform.

[0131] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separate from the processor in various aspects provided in the present disclosure, the memory can also be located inside the processor (e.g., a cache or register).

[0132] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or soon to become known to those skilled in the art are intended to be covered by the claims.

Claims

1. A method for training a target model, wherein the target model is a fast matching model, the method comprising: Training at least one reference model using a first dataset, the at least one reference model being an underlying cross-matching model, the first dataset comprising a plurality of samples, each sample comprising at least a search query, a document, and an enumerated annotation indicating a relevance between the search query and the document; Scoring the second data set and the third data set respectively using the at least one reference model to obtain a scored second data set and a scored third data set, wherein the second data set and the third data set respectively include a plurality of samples, each sample including at least a search query and a document, and the scoring includes: for each sample, scoring the relevance between the search query and the document in the sample; training the target model using the scored second dataset; and Optimizing the target model using a scored third dataset, wherein the scored third dataset includes a plurality of samples, each sample including a search query, a document, an annotation, and a target score, and the optimizing includes, for each sample: Scoring the relevance between the search query and the document in the sample using the target model to obtain a prediction score for the sample; Calculating a prediction loss corresponding to the sample based on a combination of the label and target scores in the sample and the prediction score; and The target model is optimized by minimizing the prediction loss.

2. The method according to claim 1, wherein The training of the at least one reference model comprises, for each sample: Converting the enumerated labels in the sample into a set of binary labels through a set of tasks; Creating a set of derived samples by combining the search queries and documents in the sample and the set of binary annotations; as well as The at least one reference model is trained using the set of derived samples.

3. The method according to claim 2, wherein: The set of binary labels includes positive labels indicating that the search query is relevant to the document and negative labels indicating that the search query is not relevant to the document, and The set of tasks converts the enumerated labels into positive labels or negative labels based on respective cutoff values.

4. The method according to claim 3, wherein: The value of the enumerated annotation is selected from a plurality of dependency values, and The set of tasks includes a main task and at least one auxiliary task, the cutoff value of the main task is a critical value among the multiple correlation values ​​that divides the correlation between the search query and the document into relevant and irrelevant, and the cutoff value of the at least one auxiliary task is other values ​​among the multiple correlation values ​​except the critical value.

5. The method according to claim 2, wherein: The using the set of derived samples to train the at least one reference model comprises: Scoring the set of derived samples respectively using the at least one reference model to obtain a set of prediction scores respectively corresponding to the set of derived samples; Calculating a set of prediction losses corresponding to the set of derived samples respectively based on the set of binary labels and the set of prediction scores; generating a composite predicted loss based on the set of predicted losses; and The at least one reference model is optimized by minimizing the combined prediction loss.

6. The method according to claim 1, wherein The scoring includes, for each sample: Scoring the relevance between the search query and the document in the sample using the at least one reference model to obtain at least one initial score for the sample; as well as A target score for the sample is generated based on the at least one initial score.

7. The method according to claim 1, wherein The calculating the prediction loss includes: determining the credibility of the sample based on whether the combination satisfies a predetermined rule; Setting a weight corresponding to the sample based on the credibility of the sample; and The prediction loss is calculated based on the weights.

8. The method according to claim 7, wherein: The predetermined rule at least takes the annotation as a reference.

9. The method according to claim 7, wherein: The predetermined rules include: When the annotation indicates that the search query and the document are relevant, the predicted score is greater than the target score; and When the annotation indicates that the search query and the document are not relevant, the predicted score is less than the target score.

10. The method according to claim 7, wherein: The weight is set based on a predetermined criterion including that a weight corresponding to a sample indicated as credible by the credibility is less than or equal to a weight corresponding to a sample indicated as uncredible by the credibility.

11. The method according to claim 1, wherein The at least one reference model has the same model structure or has a different model structure.

12. The method according to claim 1, wherein The second data set includes a plurality of samples based on search log data.

13. A device for training a target model, wherein the target model is a fast matching model, the device comprising: a reference model training module configured to train at least one reference model using a first data set, the at least one reference model being an underlying cross-matching model, the first data set comprising a plurality of samples, each sample comprising at least a search query, a document, and an enumerated annotation indicating a relevance between the search query and the document; a scoring module, configured to score the second data set and the third data set respectively using the at least one reference model to obtain a scored second data set and a scored third data set, wherein the second data set and the third data set respectively include a plurality of samples, each sample including at least a search query and a document, and the scoring includes: for each sample, scoring the relevance between the search query and the document in the sample; a target model training module, configured to train the target model using the scored second data set; and an optimization module configured to optimize the target model using a scored third data set, wherein the scored third data set includes a plurality of samples, each sample including a search query, a document, an annotation, and a target score, and the optimization module is further configured to, for each sample: Scoring the relevance between the search query and the document in the sample using the target model to obtain a prediction score for the sample; Calculating a prediction loss corresponding to the sample based on a combination of the label and target scores in the sample and the prediction score; and The target model is optimized by minimizing the prediction loss.

14. The device according to claim 13, wherein The reference model training module is further configured to, for each sample: Converting the enumerated labels in the sample into a set of binary labels through a set of tasks; Creating a set of derived samples by combining the search queries and documents in the sample and the set of binary annotations; as well as The at least one reference model is trained using the set of derived samples.

15. The device according to claim 13, wherein The scoring module is further configured to, for each sample: Scoring the relevance between the search query and the document in the sample using the at least one reference model to obtain at least one initial score for the sample; as well as A target score for the sample is generated based on the at least one initial score.

16. The device according to claim 13, wherein The calculating the prediction loss includes: determining the credibility of the sample based on whether the combination satisfies a predetermined rule; Setting a weight corresponding to the sample based on the credibility of the sample; and The prediction loss is calculated based on the weights.

17. A device for training a target model, wherein the target model is a fast matching model, the device comprising: at least one processor; as well as a memory storing computer-executable instructions that, when executed, cause the at least one processor to: Training at least one reference model using a first dataset, the at least one reference model being an underlying cross-matching model, the first dataset comprising a plurality of samples, each sample comprising at least a search query, a document, and an enumerated annotation indicating a relevance between the search query and the document; Scoring the second data set and the third data set respectively using the at least one reference model to obtain a scored second data set and a scored third data set, wherein the second data set and the third data set respectively include a plurality of samples, each sample including at least a search query and a document, and the scoring includes: for each sample, scoring the relevance between the search query and the document in the sample; training the target model using the scored second dataset; and Optimizing the target model using a scored third dataset, wherein the scored third dataset includes a plurality of samples, each sample including a search query, a document, an annotation, and a target score, and the optimizing includes, for each sample: Scoring the relevance between the search query and the document in the sample using the target model to obtain a prediction score for the sample; Calculating a prediction loss corresponding to the sample based on a combination of the label and target scores in the sample and the prediction score; and The target model is optimized by minimizing the prediction loss.

Images