Multi-channel recall model training method, multi-channel recall method, device and electronic equipment

Abstract

A multi-path recall model training method, a multi-path recall method, a device and an electronic device, relate to the technical field of artificial intelligence. The multi-path recall model comprises M single-path recall models. The training method comprises obtaining multiple groups of training data, each group of training data comprising a target positive sample corresponding to a search term and multiple negative samples; obtaining a relevance score of all samples in each group of training data through an M-1th single-path recall model, then performing data fitting according to the relevance score and a first loss function to determine a first expected recall result of an Mth single-path recall model, and the first loss function is used to determine the loss of the M-1th single-path recall model; determining the loss of the Mth single-path recall model according to the first expected recall result and a second loss function; and training the Mth single-path recall model with the minimum loss as the target. In this way, the Mth single-path recall model can compensate for the loss of the Mth model as much as possible to improve the overall recall effect of the multi-path recall model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-path retrieval model training method, a multi-path retrieval method, a device and an electronic device. BACKGROUND

[0002] The retrieval model is the core technology of the search engine, and mainly functions to retrieve thousands of relevant documents from a candidate set of hundreds of billions according to a search word input by a user, and input the relevant documents to a downstream precision ranking model, so as to display the relevant documents sorted according to the relevance to the user.

[0003] In related technologies, due to the certain limitations of the performance of a single-path retrieval model, the retrieval effect is poor, and therefore a multi-path retrieval mode can be used in the search engine, that is, a plurality of single-path retrieval models are used for retrieval, and then the documents retrieved by the single-path retrieval models are input to the downstream. However, since the plurality of single-path retrieval models are independent of each other, the retrieval effects of these models can have similar problems, not only wasting the model training and inference costs, but also the overall retrieval effect of the plurality of single-path retrieval models cannot be improved relative to the single model, and the overall retrieval effect is poor. SUMMARY

[0004] The present application provides a multi-path retrieval model training method, a multi-path retrieval method, a device, an electronic device, a computer storage medium and a computer program product, which can improve the retrieval effect of multi-path retrieval.

[0005] In a first aspect, the present application provides a multi-path retrieval model training method, the multi-path retrieval model comprising M single-path retrieval models, M≥2, and the method comprising:

[0006] Obtaining a plurality of sets of training data, each set of training data comprising a target positive sample corresponding to a search word and a plurality of negative samples, each sample comprising a search word and a corresponding document;

[0007] Obtaining, by the M-1th single-path retrieval model, a relevance score of all samples in each set of training data, the relevance score representing a matching degree of the document in the corresponding sample and the search word;

[0008] According to the relevance scores of the sets of training data and a first loss function, performing data fitting to determine a first expected retrieval result of the Mth model, the first loss function being used to determine a deviation between the M-1th model and a second expected retrieval result when retrieving the corresponding training data;

[0009] determine a loss of the Mth single-path recall model according to the first expected recall result and the second loss function, the loss representing a deviation between the Mth single-path recall model and the first expected recall result when recalling the corresponding training data;

[0010] train the Mth single-path recall model with the loss as a target.

[0011] In the training method provided in the present application, the M-1th single-path recall model is deviated from the second expected recall result corresponding thereto, i.e., the recall result expected to be achieved by the model, and the deviation is the loss of the single-path recall model. Therefore, in order to compensate for the loss of the M-1th single-path recall model, the M-1th single-path recall model can be used to recall each group of training data to obtain the relevance scores of the training data in the training process. Then, a second expected recall result close to the first expected recall result can be determined by using the first loss function of the M-1th single-path recall model and the obtained relevance scores, so that the recall result is used as an expected target of the Mth single-path recall model, and the Mth single-path recall model is trained with the loss as a target, so that the difference between the recall result output by the Mth single-path recall model and the first expected recall result can be as small as possible. In this way, the Mth single-path recall model trained can compensate for the loss of the M-1th single-path recall model as much as possible, so as to complement the functions of the M-1th single-path recall model, thereby improving the overall recall effect of the multi-path recall model.

[0012] In a possible implementation, after the Mth single-path recall model is trained with the loss as a target, the training method further includes:

[0013] determine a total loss of a preset neural network model according to each group of training data, the first weight, the second weight and the proxy function, wherein the preset neural network model includes M single-path recall models, the total loss represents an error between output data of the neural network model and expected output data, the first weight includes M-1 first weights, and the M-1 first weights are weights of the 1st to M-1th single-path recall models respectively, and the second weight is a weight of the Mth single-path recall model;

[0014] adjust the value of the second weight to train the neural network model in a direction of minimizing the total loss;

[0015] determine the value of the second weight corresponding to the total loss when the training is stopped as a target weight of the Mth single-path recall model.

[0016] In this way, each time a single-path recall model is trained, a target weight of the single-path recall model is determined, so that the target weights of the single-path recall models are used to balance the single-path recall results in the multi-path recall model, and effective fusion of the single-path recall results is achieved.

[0017] In a possible implementation, the value of the second weight corresponding to the total loss when the training is stopped is determined as the target weight of the Mth single-path recall model, including:

[0018] The values of the total loss in each training process of the neural network model are sequentially placed in a fixed-length sequence.

[0019] When the variable rate of the difference between the maximum value and the minimum value in the fixed-length sequence in a preset time period is lower than a preset threshold, the training is stopped, and the value of the corresponding second weight when the training is stopped is determined as the target weight.

[0020] In this way, by monitoring the change of the total loss in each training process of the neural network model through the fixed-length sequence, whether the second weight currently used in the training is a suitable weight for the Mth single-path recall model can be reflected in a timely manner, so that the target weight is quickly and efficiently determined, and the model training efficiency is improved.

[0021] In a possible implementation, the plurality of sets of training data include:

[0022] The relevance scores of a plurality of documents based on the query term recall are obtained through a preset recall model, wherein the plurality of documents come from a preset data set, the data set is a set including a plurality of documents related to the query term, and part of the documents in the set are documents labeled as positive samples;

[0023] After the plurality of documents are arranged in descending order of the relevance scores, the documents ranked in the first T positions and not labeled as positive samples are determined as first negative samples.

[0024] The plurality of first negative samples are used to form a negative sample set, so that the training data are obtained according to the negative sample set and the positive samples.

[0025] In this implementation, the relevance scores of the plurality of documents based on the query term recall can be obtained through a mature recall model capable of sparsely representing the query term, based on the data set. Then, by comparing the sizes of the relevance scores, the documents with high relevance scores but not being positive samples are selected as the first negative samples to form the training data. In this way, the documents with high relevance but not being positive samples, such as documents not interesting to users, in the collected data can be used as negative samples, so that the model training effect, such as the model training accuracy, is improved.

[0026] In some possible implementations, obtaining the plurality of sets of training data further includes:

[0027] obtain relevance scores of all documents recalled from the data set based on the search term by using the first model, wherein the data set comprises a positive sample set and a negative sample set, and the negative sample set comprises a plurality of negative samples;

[0028] determine a target sample according to the relevance scores of all the recalled documents, the target sample being a sample whose relevance score reaches a preset value and is labeled incorrectly;

[0029] obtain probability distribution data of all the negative samples according to the relevance scores of the documents corresponding to all the negative samples in the negative sample set;

[0030] select negative samples in the negative sample set that are distributed within a preset value range as second negative samples according to the probability distribution data, so that the second negative samples and the positive sample set form training data, wherein the target sample is not included in the second negative samples.

[0031] In this implementation, the relevance scores of all the documents in the data set can be obtained by using the M-1th single-path recall model to perform a recall operation based on the positive sample set and the negative sample set in the data set. Then, by processing the probability distribution of all the negative samples, documents with high relevance scores but incorrect labels, such as documents that are not correctly labeled as positive samples, can be filtered out. Negative samples that meet a preset probability value range are selected as second negative samples to form training data. In this way, the negative samples can be resampled based on the collected data, and negative samples with high relevance scores but incorrect labels can be filtered out, thereby alleviating the problem of false negative examples in the training process and improving the model training effect.

[0032] In a second aspect, the present application provides a multi-path recall method, which is implemented by using the multi-path recall model trained as described above. The multi-path recall model comprises M single-path recall models, and M≥2. The multi-path recall method comprises the following steps:

[0033] inputting the search term into each single-path recall model in the multi-path recall model to obtain a first recall result of each single-path recall model;

[0034] fusing each first recall result by using the weight corresponding to each single-path recall model to obtain a second recall result.

[0035] Since the single-path recall models in the multi-path recall model are complementary to each other, the multi-path recall model is used to perform multi-path recall, and the weight corresponding to each single-path recall model is used to fuse the corresponding recall result, thereby achieving effective balance of multi-path model joint recall.

[0036] In a third aspect, the present application provides a multi-path recall model training device. The multi-path recall model comprises M single-path recall models, and M≥2. The training device comprises:

[0037] The first obtaining model is configured to obtain a plurality of sets of training data, each set of training data comprising a target positive sample corresponding to a search term and a plurality of negative samples, each sample comprising the search term and a corresponding document;

[0038] The first processing module is configured to obtain, by the M-1th single-path recall model, a relevance score of all samples in each set of training data, the relevance score representing a matching degree of the document in the corresponding sample and the search term;

[0039] The first processing module is further configured to determine a first expected recall result of the Mth model according to the relevance scores of the sets of training data and a first loss function, the first loss function being configured to determine a deviation between the M-1th model and a second expected recall result when recalling the corresponding training data;

[0040] The first processing module is further configured to determine a loss of the Mth single-path recall model according to the first expected recall result and a second loss function, the loss representing a deviation between the Mth single-path recall model and the first expected recall result when recalling the corresponding training data;

[0041] The first processing module is further configured to train the Mth single-path recall model with the loss being minimized as a target.

[0042] In a possible implementation, the first processing module is further configured to determine a total loss of a preset neural network model according to the sets of training data, a first weight, a second weight and a proxy function, wherein the preset neural network model comprises the M single-path recall models, the total loss represents an error between output data of the neural network model and expected output data, the first weight comprises M-1 first weights, and the M-1 first weights are weights of the 1st to M-1th single-path recall models respectively, and the second weight is a weight of the Mth single-path recall model;

[0043] The first processing module is further configured to adjust a value of the second weight to train the neural network model in a direction in which the total loss is minimized;

[0044] The first processing module is further configured to determine a value of the second weight corresponding to the total loss when the training is stopped as a target weight of the Mth single-path recall model.

[0045] In a possible implementation, the first processing module is further configured to sequentially place values of the total loss in each round of training of the neural network model into a fixed-length sequence.

[0046] The first processing module is further configured to stop the training when a variable rate of a difference between a maximum value and a minimum value in the fixed-length sequence within a preset time length is lower than a preset threshold, and determine a value of the second weight corresponding to the total loss when the training is stopped as the target weight.

[0047] In a possible implementation, the first processing module is further configured to obtain relevance scores of the plurality of documents recalled based on the search term by using a preset recall model, wherein the plurality of documents are from a preset data set, the data set is a set including a plurality of documents related to the search term, and part of the documents in the set are documents labeled as positive samples;

[0048] The first processing module is further configured to arrange the plurality of documents in descending order of the relevance scores, and determine, as first negative samples, documents ranked in the first T positions and not labeled as positive samples.

[0049] The first processing module is further configured to form a negative sample set by using the first negative samples, so as to obtain training data according to the negative sample set and the positive samples.

[0050] In a possible implementation, the first processing module is further configured to obtain relevance scores of all documents recalled from the data set based on the search term by using the first model, wherein the data set includes the positive sample set and the negative sample set, and the negative sample set includes a plurality of negative samples.

[0051] The first processing module is further configured to determine, as a target sample, a sample that is labeled incorrectly according to the relevance score reaching a preset value.

[0052] The first processing module is further configured to obtain probability distribution data of all negative samples according to the relevance scores of the documents corresponding to the negative samples in the negative sample set.

[0053] The first processing module is further configured to select, as second negative samples, negative samples distributed in a preset value range in the negative sample set according to the probability distribution data, so that the second negative samples and the positive sample set form the training data, and the target sample is not included in the second negative samples.

[0054] In a fourth aspect, the present application provides a multi-path recall device, which comprises:

[0055] A second obtaining model is configured to obtain a search term.

[0056] A second processing model is configured to input the search term into each single-path recall model in the multi-path recall model to obtain a first recall result of each single-path recall model.

[0057] The second processing model is further configured to fuse the first recall results by using the fusion weights corresponding to the single-path recall models to obtain a second recall result.

[0058] In a fifth aspect, the present application provides an electronic device, comprising: at least one memory for storing a program; at least one processor for executing the program stored in the memory; wherein when the program stored in the memory is executed, the processor is configured to execute the method described in any possible implementation manner of the first aspect or the second aspect.

[0059] In a sixth aspect, the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a processor, the processor executes the method described in any possible implementation manner of the first aspect or the second aspect.

[0060] In a seventh aspect, the present application provides a computer program product, characterized by, when the computer program product is run on a processor, the processor executes the method described in any possible implementation manner of the first aspect or the second aspect.

[0061] In an eighth aspect, the present application provides a chip, characterized by comprising at least one processor and an interface; the at least one processor acquires program instructions or data through the interface; the at least one processor is configured to execute the program instructions to implement the method described in any possible implementation manner of the first aspect or the second aspect.

[0062] It can be understood that the beneficial effects of the above-mentioned second aspect to the eighth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a flowchart of a multi-path recall provided by an embodiment of the present application;

[0064] Figure 2 is a structural schematic diagram of a multi-path recall model provided by an embodiment of the present application;

[0065] Figure 3 is a flowchart of a multi-path recall model training method provided by an embodiment of the present application;

[0066] Figure 4 is a flowchart of a negative sample resampling method provided by an embodiment of the present application;

[0067] Figure 5 is a flowchart of a weight search method provided by an embodiment of the present application;

[0068] Figure 6 is a flowchart of a multi-path recall method provided by an embodiment of the present application;

[0069] Figure 7 is a flowchart of a multi-path recall method provided by an embodiment of the present application;

[0070] Figure 8 is a structural schematic diagram of a multi-path recall model training device provided by an embodiment of the present application;

[0071] Figure 9 is a structural schematic diagram of a multi-path recall device provided by an embodiment of the present application

[0072] Figure 10 is a structural schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0073] The term "and / or" in the present document is used to describe an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " in the present document represents an or relationship of associated objects, for example, A / B represents A or B.

[0074] The terms "first" and "second" and the like in the description and claims of the present document are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, rather than to describe a specific order of the response messages.

[0075] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present concepts in a particular manner. The words "exemplary" and "for example" are used in the embodiments of the present application to mean serving as an example, instance, or illustration.

[0076] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0077] First, the technical terms related to the present application are introduced.

[0078] (1) Positive example: In the field of information retrieval (IR), if a document (doc) and a query are a pair of data that are correctly matched, the pair of data is a positive example.

[0079] (2) Negative example: In the field of information retrieval (IR), if a document and a query are a pair of data that cannot be correctly matched, the pair of data is a negative example.

[0080] (3) Recall (match): refers to screening as much correct information as possible from the entire information set based on the search terms.

[0081] (4) Recall model: refers to a neural network model with recall capability.

[0082] (5) Single-way recall model: refers to a single recall model.

[0083] (6) Multi-way recall model: refers to a combined model composed of multiple single-way recall models, which can use multiple single-way recall models to perform single-way recall separately, and then fuse the results of each single-way recall and output them.

[0084] Generally speaking, commonly used recall models include sparse representation models and dense representation models. Sparse representation models offer better scalability and interpretability, but suffer from vocabulary mismatches, resulting in poor recall performance. Therefore, while sparse representation models can efficiently support indexing billions of documents for recall, they can only meet basic recall requirements, resulting in poor accuracy in matching recalled documents with search terms. While dense representation models offer better semantic matching performance for recall, they lack interpretability and are less scalable than sparse representation models. Therefore, they are only suitable for information retrieval on small datasets and are less effective for industrial-scale document retrieval.

[0085] In this regard, based on the complementary capabilities of these two models in terms of semantic matching, precise matching, efficiency, interpretability, and maintainability, a multi-way recall method can be adopted to train the sparse representation model and the dense representation model separately, and then sort the recall results output by the two models together as the final multi-way recall result. However, since the two models recall independently, this method cannot guarantee that the recall effects of the two models complement each other. Therefore, in another multi-way recall method, a complementary lexical retrieval model with semantic residual embeddings (CLEAR) can be used. Through heuristic residual design, a dense representation model is trained on the basis of an unsupervised sparse representation model, such as a similarity model (best match 25, BM25), to fit the residual between the BM25 recall result and the true recall result, so as to determine the true recall result based on the fitted residual. This residual fitting process can be expressed by the following formulas (1) and (2):

[0086] m r (Slex(q,d+ ), Slex(q, d - )) = ξ - λ train (Slex(q, d + ), Slex(q, d - )) (1)

[0087]

[0088] In formula (1) and formula (2), q represents a query, (q, d + ) represents a positive sample, (q, d - ) represents a negative sample, Slex represents the loss of BM25, Semb represents the loss when training the dense representation model, m r , ξ, and λ train are hyperparameters set when training the dense representation model, is the residual error between the recall result of BM25 and the true recall result.

[0089] The above heuristic-based residual design is only applicable to the above combination model and lacks universality and effectiveness. Moreover, this method is a local optimization for the recall model and can only optimize the dense representation model, and cannot achieve global optimization, so the effect is significantly reduced on a large-scale data set and is difficult to apply to an actual engine.

[0090] In order to improve the recall effect of the recall model, the embodiment of the present application provides a multi-path recall model training method, a multi-path recall method, a device and electronic equipment. The multi-path recall model training method is a method for training a multi-path recall model based on boosting theory. The boosting theory is the additive boosting theory, which optimizes each single-path recall model in the training round based on the previous single-path recall model, and reduces the deviation of the entire model recall result through iteration. Moreover, each single-path recall model in the training round is a model complementary to the recall effect of the previous single-path recall model, so that the final trained multi-path recall model includes at least two models, and the recall effects of the at least two models are complementary. In addition, the multi-path recall method of the embodiment of the present application can use the final trained multi-path recall model to perform multi-path recall, and balance the recall results of the single-path recall models through the fusion weights corresponding to the single-path recall models in the multi-path recall process, so as to effectively fuse the multi-path recall results and effectively improve the overall recall effect of the multi-path recall model.

[0091] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the following first describes some scenarios in which the multi-path recall model is used for multi-path recall in the embodiments of the present application.

[0092] For example, the multi-way recall model in the embodiments of the present application can be applied to a machine learning-based search engine recall scenario. The general process for performing multi-way recall in a search engine is as follows: the user enters a desired query, and the search engine uses each single-way recall model to search for results in multiple indexes. The results from each index are then fused and filtered to obtain relevant documents, which are then fed into downstream tasks.

[0093] For example, the multi-channel recall model can be specifically applied to search and recall scenarios within large-scale searches, information streams, service platforms, or applications (apps), where:

[0094] In the big search application scenario, big search is to use keyword search to display relevant information. For example, according to the user's search query, match the web page information that the user may be interested in and display it. For example, Figure 1 As shown in the flowchart, web page data collected from the Internet is formed into a data set through S1, and after data preprocessing through S2, an index (indexer) of the data set is created through S3. When a user enters a keyword (i.e., a search term, the same below) through the search box provided by the search engine page, such as "patent application", the search engine's multi-way recall model executes S4 to obtain the keyword, and then performs multi-way recall through S5 to access the data set using the indexer, and recall web page information related to the keyword "patent application" from the data set, such as official website information of patent application, patent search website information, patent agency website information, etc. The multi-way recall model also marks the relevance score for the recalled web page information, and then, after fusion through S6, it is transmitted to the downstream refined ranking model, and all recalled web page information is sorted according to the corresponding relevance score through S7, and the sorting results are output to the search page for display through S8.

[0095] In information flow applications, information flow utilizes user preferences and hobbies to display relevant information. For example, if a user's preference information includes "music," the multi-way recall model can retrieve and display data about songs, instruments, and concerts from the database based on the information that other users with the same music preference are interested in, such as songs, instruments, and concerts.

[0096] In the application scenario of a service platform, the service platform may include a platform dedicated to providing specific services, such as an Internet resource search platform, a shopping platform, an online video media platform, etc. The service platform can collect users' keywords or preference features and, through a multi-channel recall model, recall relevant data from the corresponding database for display. For example, if a user's historical input keywords on the shopping platform include "backpack," the multi-channel recall model can be used to recall several shopping web pages related to backpacks and display them to the user. Alternatively, if a user enters keywords such as TV series titles or character names on a web page provided by an online video media platform, the corresponding multi-channel recall model can be used to recall video or audio web pages related to the keywords and display them to the user.

[0097] In the in-app search scenario, apps can include application market programs, video apps, music apps, news apps, browsers, etc. deployed on the terminal. These apps can collect user keywords or preference features and recall relevant data from the corresponding database through a multi-channel recall model for display.

[0098] In some examples, the multi-way recall model can be applied to business systems such as question-answering systems and recommendation systems to recall relevant data information based on keywords or preference features entered by users.

[0099] For example, Figure 2 The structure of a multi-way recall model is shown. It is understood that the model can be deployed in any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 The multi-way recall model 200 shown includes a first model 210 and a second model 220, wherein the second model 220 is a complementary model obtained by training the first model 210 based on the loss of the first model 210, wherein the loss of the first model 210 represents the deviation between the recall result of the first model 210 and the expected recall result.

[0100] In this embodiment, the first model 210 may be an independent single-way recall model, capable of performing a single-way recall based on an input search term.

[0101] Exemplarily, the first model 210 can be a sparse representation model or a dense representation model, and the type of the first model 210 is not limited in the embodiment. For example, the first model 210 can be a dense representation model. In this example, the first model 210 can include an input layer 211, a representation layer 212, and a matching layer 213. The input layer 211, which can also be referred to as a token embedding layer, is configured to map text to a low-dimensional vector space to convert the text into a word vector, and provide the word vector to the representation layer 212. The text includes a search term and a document in a document library. The representation layer 212 is configured to represent the word vector by a neural network, i.e., construct a representation from a word to a sentence. The matching layer 213 is configured to perform semantic matching on the vectors output by the representation layer 212 (including a search corresponding representation vector and a document corresponding representation vector), and score the relevance between the search term and the matched document to obtain a corresponding relevance score. The relevance is the similarity between the search term and the document, and therefore, the relevance score represents the matching degree between the search term and the document. The scoring can be achieved by calculating the cosine similarity or Euclidean similarity between the search corresponding representation vector and the document corresponding representation vector, or by inner product operation.

[0102] In the embodiment, the second model 220 can be a model of the same type as the first model 210, or a model of a different type from the first model 210, which is not limited in the embodiment. For example, the first model 210 is a sparse representation model, and the complementary second model 220 can be a sparse representation model or a dense representation model. The second model 220 can be trained according to the loss of the first model 210 to form a complementary model of the first model 210. In the embodiment, the complementary model refers to a model that is complementary to the recall effect of the previous single-path recall model.

[0103] The recall effect is complementary, i.e., the currently trained single-path recall model can achieve a recall effect that the previous single-path recall model does not achieve. For example, the accuracy of the recall result of the first model 210 is 90%, and after the second model 220 is trained based on the loss of the first model 210, the accuracy can reach 95%, which is higher than that of the first model 210. Therefore, in the process of using the first model 210 and the second model 220 for multi-path recall, the recall effect of the overall multi-path recall model is improved.

[0104] In this embodiment, the loss of the first model 210 refers to the deviation between the model when recalling documents and the expected recall result thereof. Based on the loss of the first model 210, a first expected recall result that can compensate for the loss as much as possible is obtained through data fitting, and the second model 220 is trained towards the direction of being close to the first expected recall result. Therefore, the second model 220 after training can realize optimization of the recall effect on the basis of the first model 210, and ensure that the second model 220 is a model that is complementary to the recall effect of the first model 210. In this way, when the first model 210 and the second model 220 are used for joint multi-path recall, the situation that the recall effect of the second model 220 is similar to that of the first model 210 can also be avoided, and the waste of model training and inference costs can be avoided.

[0105] In one specific example, one or more single-path recall models can be continuously trained on the basis of the multi-path recall model shown in Figure 2 Specifically, the loss of the output result of the first model can be used to guide the training of the second model, so as to ensure that the second model obtained through training is necessarily a complementary model of the first model. Similarly, the third model can also be trained based on the loss of the output result of the second model. In this way, the third model obtained through training is actually based on the first model and the second model, that is, the third model is a complementary model of the sum of the first model and the second model. Therefore, it can be understood that when the multi-path recall model in this embodiment includes M models, the Mth model is a complementary model of the first M-1 models. In this way, all single-path recall models in the multi-path recall model are optimized in a step-by-step manner, which can improve the recall effect of the overall model.

[0106] Moreover, it can be understood that in the embodiments of the present application, the second model 220 as a complementary model of the first model 210 is trained based on the loss of the first model 210, and therefore is strongly related to only the corresponding loss function. In some specific examples, the internal structure and internal characteristics of the second model can not be considered, and the second model can be directly trained according to the loss of the first model 210 through black box testing.

[0107] The above is an introduction to the multi-path recall model provided by the embodiments of the present application. Through the multi-path recall model, multi-path recall can be performed.

[0108] Next, based on the above description, a multi-path recall model training method provided by the embodiments of the present application is introduced. It can be understood that the method is proposed based on the above description, and part or all of the contents in the method can be referred to in the above description.

[0109] Please refer to Figure 3 , Figure 3is a flowchart of a multi-recall model training method provided by an embodiment of the present application. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities.

[0110] As shown in Figure 3 , the multi-recall model training method can include S301-S306:

[0111] S301, obtaining a first training set, the first training set including a positive sample set and a negative sample set.

[0112] In the embodiment, the positive sample set and the negative sample set are both subsets of the first training set, and each subset includes a plurality of samples.

[0113] In the embodiment, the positive sample set and the negative sample set can be, but are not limited to, collected by a search engine, and the collected positive sample set and negative sample set form the first training set.

[0114] Among them, for the positive sample set, it can be obtained by the following way:

[0115] For example, the search engine can collect the search terms input by the user and the corresponding web pages clicked by the user to obtain the positive sample set. For example, the search term input by the user to the search engine is "leaves", and through the existing recall model in the search engine, 500 web pages related to leaves are recalled, and among the 500 web pages, 80 web pages are historically clicked by the user, then the 80 web pages can be respectively corresponding to the search term "leaves", forming 80 data pairs (query, doc), each data pair is labeled as a positive sample, forming a positive sample set containing 80 positive samples, which is used for subsequent model training.

[0116] It can be understood that in the embodiment, the positive sample set obtained can also include data pairs composed of multiple search terms and corresponding documents. For example, the positive sample set includes data pairs (search term Q1, document D1), (search term Q2, document D2), (search term Q3, document D3), (search term Q4, document D4), and so on.

[0117] In the embodiment, for the negative sample set, it can be collected by any one or several of the following ways:

[0118] Method one:

[0119] Exemplarily, the negative samples can be randomly sampled from a full document library, where the full document library is a database having a plurality of documents, and all the documents in the database have an index relationship with a plurality of search terms, so that the single-pass recall model can find the relevant documents in the database based on the search terms and the corresponding index relationship. For example, there are 50 million web pages in the full document library of a search engine, and 1000 web pages have an index relationship with the search term "leaves". 40 web pages can be randomly sampled from the 50 million web pages as negative samples corresponding to the search term "leaves" to form a negative sample set.

[0120] It can be understood that the negative sample set is constructed corresponding to the search terms of the positive sample set, for example, the positive sample set includes positive samples corresponding to 10 search terms, and the negative sample set also includes negative samples corresponding to the 10 search terms. In some specific examples, the documents as negative samples corresponding to different search terms can be the same, for example, the search term Q1 and the search term Q2 both have 10 negative samples, and the 10 documents corresponding to the negative samples of the search term Q1 can be the same as the 10 documents corresponding to the negative samples of the search term Q2. The following several ways of obtaining the negative sample set are the same, and the obtained negative sample set is constructed corresponding to the search terms of the positive sample set.

[0121] Method two:

[0122] Exemplarily, the negative sample set can also be formed by sampling once, and the documents not clicked by the user among the documents recalled by the single-pass recall model according to the search term are taken as negative samples. For example, when the search term is "leaves", the search engine recalls 500 relevant web pages, and randomly selects 40 web pages not clicked by the user as negative samples to construct a negative sample set.

[0123] Method three:

[0124] In one possible implementation, the negative sample set can be obtained by the following S3011 to S3014, wherein the negative sample set includes a plurality of first negative samples:

[0125] S3011, the relevance scores of the documents recalled based on the search term are obtained by scoring the relevance of the documents recalled by a separate recall model, wherein the recalled documents come from a preset data set, and the data set is a data set composed of a plurality of documents related to the search term, and part of the documents in the data set are documents labeled as positive samples.

[0126] In this embodiment, a dataset of documents related to the query term is collected in advance, and part of the documents in the dataset are labeled as positive samples. For example, a search engine can be used to collect a query term input by a user and the corresponding retrieved web pages to build a dataset. For example, based on the query term Q input by the user to the search engine, a total of 500 relevant web pages are retrieved, forming a dataset, and 80 web pages in the dataset that have been historically clicked by the user are labeled as positive samples as a positive sample set.

[0127] After the dataset is built, the query term is input into a simple retrieval model, which is used to retrieve and score the relevance of all relevant documents in the dataset based on the query term, to obtain the relevance scores of these documents. The retrieval model can be any type or structure and can be a sparse representation of the query term, such as a BM25 model.

[0128] For example, the first model shown in Figure 2 For example, the first model shown in Figure 2 As shown in FIG. 21, after the query term query is input into the first model 210, the query and the documents in the dataset are all divided into at least one token in the input layer 211, which can be denoted as Toki, i = 1, 2, 3, … i, …, N. And by setting the start identifier cls and the end identifier sep, Toki belonging to the same word is marked. All tokens Toki are converted into word vectors through token embedding processing, and each word vector can represent certain semantic information of the word it belongs to. Then the word vectors corresponding to the query term and the documents are built into representation vectors such as sentences by the representation layer 212. The representation vectors are input into the matching layer 213 for semantic matching, and the similarity (such as cosine similarity) between the representation vector of the query term and the representation vector of each document is calculated during the matching process to score the relevance, and the similarity calculation result is the relevance score of each document.

[0129] S3012, the documents retrieved in S3011 are sorted in descending order of their relevance scores, and the top T documents that are not labeled as positive samples are determined as the first negative samples.

[0130] In this embodiment, after obtaining the relevance scores of the retrieved documents by S3011, the retrieved documents can be sorted according to the order of their relevance scores from high to low. Then the documents with high relevance but not positive samples in the sorted results are used as the first negative samples. Wherein, T can be set in advance according to actual sampling requirements, and T is a positive integer, such as T = 50.

[0131] For example, among the 500 web pages included in the data set, 80 web pages are labeled as positive samples. After obtaining the relevance scores of the 500 web pages through S3011, the web pages are sorted in descending order of the relevance scores, and then the top 20 web pages that are not positive samples are screened out as the first negative samples.

[0132] It can be understood that in the process of obtaining the first negative samples, when the documents are sorted, the documents can also be sorted in ascending order of the relevance scores. At this time, the Tth ranked documents that are not labeled as positive samples can be used as the first negative samples.

[0133] In this way, the documents with high relevance but not positive samples (for example, samples with high relevance but not interesting to users) can be screened out as the first negative samples, and form a static first training set together with the positive sample set for subsequent training of the complementary model, which helps to improve the accuracy of the complementary model.

[0134] Optionally, in some examples, if the number of first negative samples collected is small, the first negative samples can be combined with the negative samples obtained through the first or second method to form a negative sample set, and the negative sample set and the positive sample set obtained through any of the above methods can be used as the first training set for subsequent training of the complementary model.

[0135] In the embodiments of the present application, the first training set can be obtained after obtaining the positive sample set and the negative sample set through any one or more of the above methods. In some specific examples, the first training set can be directly used for subsequent training of the complementary model.

[0136] In other specific examples, to improve the training effect of the complementary model, the negative samples can also be resampled based on the first training set or other pre-constructed training sets to form a new training set for training of the complementary model.

[0137] Next, taking resampling of negative samples based on the first training set and forming a new training set, i.e., a second training set, as an example, the resampling of negative samples will be described. Referring to FIG. 4, the resampling of negative samples in this example can include S401 to S404: Figure 4

[0138] S401, using the first model, performing relevance scoring on all documents in the first training set to obtain the relevance scores of the corresponding documents.

[0139] In this embodiment, referring to FIG. 4, S401 can include S4011 and S4012: Figure 2 ​As shown, all documents in the positive sample set and the negative sample set of the first training set are taken as a data set, the search terms corresponding to the positive sample set and the negative sample set are input into the first model 210, all documents are recalled from the data set and the relevance of the documents is scored to obtain the relevance scores of the documents, that is, the relevance scores corresponding to all samples in the positive sample set and the negative sample set. The process of scoring the relevance of all documents in the positive sample set and the negative sample set by the first model 210 can refer to the process of scoring the recalled documents by the single-path recall model in S011 above, which will not be described here.

[0140] S402, according to the relevance scores, determining target samples, the target samples being samples whose corresponding documents are ranked in the top m positions and are labeled incorrectly after all documents are sorted in descending order of the relevance scores.

[0141] In this embodiment, all samples in the first training set are sorted in ascending order of the relevance scores of the corresponding documents, and the documents with higher relevance are arranged in front positions, and vice versa. Then, based on all sorted documents, the documents whose corresponding documents are ranked in the top m positions and are labeled incorrectly are determined, wherein the incorrectly labeled documents are samples with high relevance scores but incorrectly labeled as positive samples, or samples with high relevance scores but incorrectly labeled as negative samples. Then, these determined incorrectly labeled samples are taken as target samples. Wherein, m is a positive integer.

[0142] S403, according to the relevance scores of all documents in the negative sample set, obtaining the probability distribution data of all negative samples in the negative sample set.

[0143] In this embodiment, the relevance scores of the documents corresponding to each search term in the negative sample set are mapped to probability values in a preset interval, and according to the probability values of all documents in the negative sample set, the probability distribution data of all documents in the negative sample set in a specific interval can be obtained, that is, the probability distribution data of all negative samples in the negative sample set.

[0144] For example, the relevance scores of all documents in the negative sample set can be mapped to the [0, 1] space by a normalization exponential function (softmax) to obtain the probability values of all documents in the negative sample set, and the sum of the probability values of all documents in the negative sample set is 1. In this way, according to the probability values of each document in the negative sample set, the distribution of these documents in the [0, 1] space is determined, so that the probability distribution of all negative samples in the negative sample set can be intuitively reflected.

[0145] S404, according to the probability distribution data, selecting negative samples in the negative sample set distributed in a preset value range as second negative samples, so that the second negative samples and the positive sample set form a second training set, wherein the target samples are not included in the second negative samples.

[0146] In this embodiment, after the probability distribution mapping processing is performed on the documents corresponding to each search term in the negative sample set, the negative samples that are more conducive to training can be screened from the obtained probability distribution of all negative samples. For example, a probability value range can be preset, and then the second negative samples are obtained by resampling according to the value range, and the target samples, i.e., the samples with high relevance scores but with annotation errors, are not included in the obtained second negative samples.

[0147] For example, there are 40 negative samples in the negative sample set, and the probability distribution data of the 40 negative samples in the negative sample set is obtained by mapping using the softmax function, and the maximum probability value is 0.7 and the minimum probability value is 0.01. Since the higher the relevance score of a sample, the higher the probability value, and the higher the possibility of being selected by resampling, and vice versa, the preset value range can be [0.05, 0.2], and the negative samples distributed in the value range are used as the second negative samples, so as to filter out the negative samples with high relevance scores but with annotation errors.

[0148] This is because there may be some false negative examples in the negative sample set. The false negative example sample refers to a sample whose document can be correctly matched with the search term in theory, but is not correctly annotated as a positive sample in practice, and the relevance score of the false negative example sample is also often high, which will adversely affect the training effect when used for model training. Therefore, in this embodiment, based on the probability distribution of the negative samples in the negative sample set, the probability value range is preset for resampling, and the false negative example samples with high relevance scores can be filtered out from all negative samples as second negative samples, so as to filter out the samples that may be false negative examples.

[0149] In this way, the second negative sample set is constructed using the second negative samples, and forms a second training set with the positive sample set, which is used for subsequent training of the complementary model, can effectively alleviate the problem of false negative examples in the subsequent training process of the complementary model, reduce the loss of the trained complementary model, and improve the training effect of the model.

[0150] It can be understood that the preset value range can be determined according to actual training requirements to ensure that the number of the second negative samples obtained by resampling meets the training requirements.

[0151] In the embodiments of the present application, the positive sample set in the first training set obtained by the foregoing S301 or in the second training set obtained by the foregoing S302 includes a plurality of positive samples. Each positive sample in the positive sample set can be taken as a group of training data D, q represents a search term, d + represents the document of the positive sample corresponding to q, d -wherein q represents the number of negative samples, and L represents the number of negative samples. In this way, a plurality of sets of training data can be obtained, and then based on each set of training data, the second model is trained iteratively based on the loss of the first model through S302 to S305, so that the second model and the first model are complementary in recall effect.

[0152] The execution process of S302 to S305 is described below with any one set of training data D as an example.

[0153] S302, the first model is used to score the relevance of all documents corresponding to the target positive sample and the negative sample set, to obtain a first relevance score and L second relevance scores, wherein the target positive sample is one of the positive samples in the positive sample set, the first relevance score represents the matching degree of the target positive sample and the search term, and the negative sample set includes L negative samples, and each second relevance score is used to represent the matching degree of the corresponding negative sample and the search term.

[0154] In the embodiment, any positive sample in the positive sample set in the first training set or the second training set obtained above is taken as an example, and the processing process of a set of training data formed by the target positive sample and the negative sample set in the first model 210 is described with reference to the first model 210 shown in Figure 2

[0155] For example, all documents of the target positive sample and the negative sample set are taken as a data set, and the search term corresponding to the target positive sample and the negative sample set is input into the input layer 211 of the first model 210. The input layer 211 performs word embedding processing to convert the search term text and the data set text into a word vector form, and then the word vector form is input into the representation layer 212 for vector representation. Next, the search term representation vector output by the representation layer 212 and the representation vector of each document in the data set are input into the matching layer 213 for semantic matching. By calculating the similarity between the search term representation vector and the representation vector of each document in the data set, the relevance score of each document with respect to the search term is obtained, that is, the first relevance score of the target positive sample and the second relevance score of each negative sample are obtained.

[0156] S303, data fitting is performed according to the first relevance score and all second relevance scores, and a first loss function, to determine the first expected recall result of the second model, and the first loss function is used to determine the deviation between the recall of the first model for any positive sample and the second expected recall result.

[0157] ​In this embodiment, the recall result obtained by the first model 210 when recalling any positive sample in the positive sample set is often deviated from the second expected recall result, i.e., the recall result that the first model is expected to achieve. The deviation is the loss of the first model when recalling the corresponding positive sample (hereinafter referred to as "the loss of the first model"). Therefore, in order to improve the overall recall effect of the multi-path recall model, a second model can be trained to make the second model compensate for the loss of the first model as much as possible, so that the recall effects of the second model and the first model are complementary, and the overall recall effect is improved.

[0158] For example, in order to make the obtained second model compensate for the loss of the first model, in this example, based on the principle of residual-based loss, the gap (i.e., residual) between the recall result of the first model 210 and the second expected recall result can be used to obtain the first expected recall result close to the second expected recall result through data fitting, and the first expected recall result is the recall result that the expected second model 220 can achieve. In this way, in the process of training the second model, by making the gap between the recall result output by the second model 220 and the first expected recall result as small as possible, the second model 220 obtained by training can compensate for the loss of the first model 210 as much as possible when performing multi-path recall with the first model 210, and the optimization of the first model is realized.

[0159] The loss of the first model 210 can be determined by the first loss function. In this way, when determining the first expected recall result, the first loss function of the first model 210 can be used to construct a fitting function, and then the first correlation score and all second correlation scores are substituted into the fitting function to obtain the first expected recall result close to the second expected recall result.

[0160] It can be understood that the first loss function can be determined according to the recall task of the first model 210, which is not limited in this embodiment. For example, the first loss function can adopt a negative log-likelyhood loss function, a binary cross entropy loss function, or a hinge loss function, etc.

[0161] For example, taking the negative log-likelyhood function as an example, the loss of the first model can be represented as:

[0162]

[0163] In formula (3), M is the label of the single-path recall model currently trained, and then denotes the loss of the single-path recall model with the label M-1, i.e., M is the label of the second model 220, is the loss of the first model 210. s(q, d + ) denotes the first relevance score of the target positive sample (q, d + ) and the second expected recall result. denotes the similarity between the relevance score of the corresponding l-th negative sample in the negative sample set and the second expected recall result, l = 1, 2, …, L, and l is a positive integer. Wherein, s(q, d + ) and The similarity calculated by both can be cosine similarity or Euclidean similarity, etc. The present example does not make unique limitation.

[0164] Exemplarily, the fitting function is constructed by using the first loss function of the first model 210 to determine the second expected recall result, i.e. when determining the fitting target in the training process of the second model, the first relevance score of the first model on each positive sample can be derived by using the loss of the first model on each positive sample to obtain the second expected recall result. Specifically, the fitting function can be represented by the following formula (4):

[0165]

[0166] In formula (4), is the expected recall result corresponding to the current trained M-th single-path recall model, F M-1 (q, d + ) denotes the relevance score output by the M-1-th single-path recall model based on the target positive sample (q, d + ). That is, is the first expected recall result of the second model, F M-1 (q, d + ) is the first relevance score and the second relevance score output by the first model based on the target positive sample (q, d + ).

[0167] In this way, the first relevance score and the second relevance score are brought into the fitting function represented by formula (4), and the first expected recall result can be obtained.

[0168] In the present example, still taking the negative log-likelihood function represented by the aforementioned formula (3) as the first loss function as an example for illustration. In formula (3), can be converted to Wherein, P M-1 (q, d + ) denotes the distribution probability of the first relevance score of the target positive sample (q, d + ) relative to the relevance scores of all samples in the current group of training data, and the target positive sample (q, d+ The first correlation score of each sample is output by the M-1th single-path recall model. For example, the distribution probability of the correlation score of each sample can be mapped to the [0, 1] space by a softmax function.

[0169] Therefore, the formula (4) is derived by substituting the formula (3) into the formula (4), and the formula (4) is derived.

[0170]

[0171] In the formula (5), the correlation score of the lth negative sample output by the M-1th single-path recall model is denoted as . Thus, it can be derived that when the second model is denoted as M, the function of the first expected recall result of the second model can also be represented as Then, the first correlation score and the L second correlation scores are mapped to a probability distribution by a softmax function to obtain P M-1 (q, d + ), so as to quickly determine the first expected recall result.

[0172] In S304, the loss of the second model is determined according to the first expected recall result and the second loss function, and the loss represents the deviation between the recall target positive sample of the second model and the first expected recall result.

[0173] In this embodiment, the second loss function is a function for calculating the loss of the second model. It can be understood that the second loss function can be determined according to the recall task of the second model, and this embodiment does not make specific limitations. For example, the second loss function can adopt a negative log-likelyhood loss function, a binary cross entropy loss function, or a hinge loss function.

[0174] In this embodiment, the second expected recall result can be used as the weight of the second loss function to determine the loss of the second model.

[0175] For example, taking the negative log-likelyhood function as the second loss function, according to the transformation principle of the formula (3) described above, it can be known that the second loss function can be represented as: wherein, P M (q, d + ) is the distribution probability of the correlation score of the first correlation score with respect to all samples in the current group of training data, and the first correlation score is output by the Mth single-path recall model based on the target positive sample. In this example, the second model is denoted as M. ​

[0176] the loss of the second model The relationship between the second expected recall result and the second model can be represented by the following formula (6):

[0177]

[0178] S305, training the second model with the minimum loss of the second model as the goal.

[0179] In this embodiment, after determining the loss of the second model, the current group of training data composed of the target positive sample and the negative sample set is input into the second model, and training is performed with the minimum loss of the second model as the goal.

[0180] For example, it is assumed that the positive sample set of the second training set includes K positive samples, each of which is sequentially taken as a target positive sample, and then forms K groups of training data D N , k = 1, 2, …, K, K is a positive integer. When initialized, the second loss function of the second model is Based on the first group of training data D1, a round of steps S302 to S304 is performed, and then the first group of training data D1 is input into the second model, and the loss obtained in this round S304 is taken as the goal for the first round of training. After the first round of training, based on the second group of training data D2, a round of steps S302 to S304 is performed again, and then the second group of training data D2 is input into the second model, and the loss obtained in this round S304 is taken as the goal for the second round of training, and so on, until all the training data is used up or the training result reaches a preset condition (such as a preset number of times), the training is ended, and at this time the second model complementary to the first model is obtained.

[0181] In this embodiment, since the training of the second model is based on the loss of the first model, the second model obtained by training is necessarily a functional complementary model of the first model.

[0182] In some possible implementations, after obtaining the first training set, a plurality of first training data can be generated, and based on the plurality of first training data, S302 to S305 can be iteratively executed to train the preset neural network to obtain a warm-up model. In this example, the warm-up model is the preliminary training result in the process of training the second model. Next, after obtaining the second training set, a plurality of second training data can be generated based on the second training set. Then, based on the plurality of second training data, S302 to S305 can be iteratively executed to continue training the warm-up model until the iteration is ended, and the second model complementary to the first model in function is obtained.

[0183] In some possible implementations, if the overall recall effect of the multi-path recall model constituted by the first model and the second model does not reach an expected target after the second model is trained, a third model can be continuously trained based on the loss of the second model, and so on, until the entire recall result of all single-path recall models reaches an expected target, such as an expected accuracy, after the nth model is trained. The embodiments of the present application are not specifically limited.

[0184] In the embodiments of the present application, if the multi-path recall model includes M single-path recall models during the training process, the Mth single-path recall model is trained based on the first M-1 single-path recall models, for example, the second model is trained based on the first model. Since the multi-path recall model is used to fuse the results of the single-path recall of each single-path recall model to achieve joint recall, in order to ensure the effectiveness of the recall result fusion, in the embodiments of the present application, after each single-path recall model is trained, the weight of the Mth single-path recall model is trained based on the back propagation algorithm through the preset neural network model, so that the Mth single-path recall model is fused with the recall results of the first M-1 single-path recall models based on the weight.

[0185] For example, referring to FIG. 6, after the Mth single-path recall model is trained, the method can further include S501-S506. Figure 5

[0186] S501, obtaining a sample training set, the sample training set including G groups of sample data, each group of sample data including one positive sample and a plurality of negative samples.

[0187] In the embodiments of the present application, the sample training set can be the first training set, the second training set or other pre-constructed training set obtained above. The sample training set includes a positive sample set and a negative sample set, wherein the positive sample set includes G positive samples, and the negative sample set includes L negative samples. The G positive samples in the positive sample set are respectively taken as target positive samples to form G groups of sample data Dd, q represents a search term, d + d represents a document of a positive sample, - d represents a document of a negative sample, and L represents the number of negative samples.

[0188] S502, determining a total loss of a preset neural network model according to the sample data, the first weight, the second weight and the proxy function, wherein the preset neural network model includes M single-path recall models, the total loss represents an error between output data of the preset neural network model and expected output data, the first weight includes M-1 first weights, the M-1 first weights are respectively weights of the first single-path recall model to the M-1th single-path recall model, and the second weight is a weight of the Mth single-path recall model. ​

[0189] In this embodiment, the preset neural network model is a neural network model adapted to the aforementioned multi-path recall model. For each single-path recall model trained in the multi-path recall model, the preset neural network model determines a corresponding weight for the single-path recall model. For example, when the multi-path recall model only includes the aforementioned first model, the preset neural network model determines the weight β1 of the first model. When the multi-path recall model includes the aforementioned second model, the preset neural network model determines the weight β2 of the second model. Therefore, when the multi-path recall model is trained to the Mth single-path recall model, the weights of the first M-1 single-path recall models, i.e., the first weights, are known parameters. The preset neural network model needs to train to obtain the weight of the Mth single-path recall model.

[0190] For example, when the multi-path recall model trains the Mth single-path recall model based on the first M-1 single-path recall models, the preset neural network model is also a network model including the M-1 single-path recall models and the Mth single-path recall model. In the preset neural network model, the M-1 single-path recall models each have a corresponding first weight, and the Mth single-path recall model has an initialized weight, i.e., the second weight has an initial value.

[0191] In this example, the purpose of training by the preset neural network model is to determine the appropriate weight or optimal weight of the Mth single-path recall model. Therefore, when starting the first round of training, the preset neural network model first configures an initial value as the second weight, and then can determine the total loss of the preset neural network model based on the first set of sample data, the first weights, the second weight, and the proxy function.

[0192] Since the role of each weight is to balance the recall result of its corresponding single-path recall model in the multi-path recall model, the final recall result of each single-path recall model is the product of the output result and the weight. Therefore, in this example, the preset neural network model takes the first set of sample data as input, and based on the first M-1 single-path recall models and the first weights, and the Mth single-path recall model and the second weight, the output result obtained is the sum of the products of the recall results of the M single-path recall models and their respective weights. Then, the error between the output data of the preset neural network model and the expected output data, i.e., the total loss, is the error between the sum and the expected output data, which can be represented by the proxy function h(β) as follows:

[0193]

[0194] In formula (7), h(β) represents the value of the proxy function (proxy h(β)); β M-1 is the weight of the M-1th single-path recall model, is the sum of the recall results of the first M-1 models, fM (q, d) is the recall result of the Mth single-way recall model, β M is the weight of the Mth single-way recall model.

[0195] S503: Adjust the value of the second weight and train the preset neural network model with the goal of minimizing the total loss.

[0196] In this embodiment, the remaining G-1 groups of sample data are sequentially input into the preset neural network model for batch training. During the training process, the second weight β is adjusted once for each round of training through the back propagation algorithm. M The value of is used to change the total loss of the neural network model so that the model iterates in the direction of minimum total loss.

[0197] For example, β can be pre-built M A data set that includes multiple values. Through the weight search algorithm, each time a round of training is performed, a value is called from the training set as β M , participating in this round of training. The multiple values ​​in the dataset can be arranged in descending order. When the preset neural network model begins training, it can call the first value in the dataset that is larger than the initial value and perform forward propagation. If the total loss value obtained at this time increases, it will then perform backward propagation and call the second value in the dataset that is smaller than the first value to participate in training to reduce the total loss. This process is repeated, and the values ​​in the dataset are continuously called for training in the direction of minimizing the total loss until the iteration ends.

[0198] S504: Determine the target weight according to the total loss value corresponding to when training is stopped.

[0199] In this embodiment, each round of batch training of the neural network model will obtain a value of h(β), that is, the total loss value. When the neural network model iteration reaches the preset condition, the training stops. At this time, the target weight can be determined based on the total loss value obtained when the training stops.

[0200] Exemplarily, the process of determining the target fusion weight may specifically include S5041 to S5042:

[0201] S5041: Put the total loss value corresponding to each round of training into a fixed-length sequence in an orderly manner.

[0202] In this example, the fixed-length sequence is a queue with a fixed length. The values of the total loss h(β) are sequentially put into the fixed-length queue in the order in which they are generated. When the number of values of h(β) in the fixed-length sequence is saturated, the value that was put into the fixed-length sequence first is removed according to the first-in first-out rule, and the latest value is put into the end of the fixed-length sequence. In this way, the latest data can be kept in the fixed-length sequence and the number of values is fixed.

[0203] S5042, when the variable rate of the difference between the maximum value and the minimum value in the fixed-length sequence is lower than the preset threshold within the preset time length, stop training, and determine the second weight at the time of stopping training as the target weight.

[0204] In this example, the values of the second weight used in each training process are monitored and recorded, and the variable rate of the difference between the maximum value and the minimum value in the fixed-length sequence is recorded. When the variable rate is lower than the preset threshold, such as 0, the iteration is stopped, and the value of the second weight corresponding to the time when the iteration is stopped is determined as the target weight. In this way, the target weight is used to balance the complementary models trained in the multi-path recall model, which is beneficial to the effective fusion of the recall results of each single-path recall model.

[0205] In some other examples, the training can also be stopped when the variable rate of the average of the maximum value and the minimum value in the fixed-length sequence is lower than the preset threshold, and the second weight at the time of stopping training is determined as the target weight. Alternatively, the training can be stopped when the sample data is exhausted or the number of iterations reaches the preset number, and the second weight at the time of stopping training is determined as the target weight.

[0206] In the above multi-path recall model training method, the samples used are all samples corresponding to search terms, and thus a multi-path recall model based on search terms can be obtained, and the single-path recall models in the multi-path recall model are complementary. It can be understood that when training a multi-path recall model based on multimedia data such as pictures and audio for recall, the same technical concept as the above multi-path recall model training method can be used to train with corresponding multimedia data as samples. For example, using picture-form samples to train a multi-path recall model based on pictures for recall, and the single-path recall models in the multi-path recall model are complementary, or using audio-form samples to train a multi-path recall model based on audio for recall, and the single-path recall models in the multi-path recall model are complementary.

[0207] Next, based on the above description, a multi-path recall method provided by an embodiment of the present application is introduced. It can be understood that this method is based on the above description, and part or all of the contents in this method can be referred to in the above description.

[0208] Figure 6A flowchart of a multi-path recall method provided by an embodiment of the present application is shown. It can be understood that the method can be based on Figure 2 implemented by the multi-path recall model, and the method can be executed by any device, equipment, platform, or cluster of equipment having computing and processing capabilities. As shown in the figure, the multi-path recall method can include S601-S602:

[0209] S601, input the search term into each single-path recall model in the multi-path recall model to obtain a first recall result of each single-path recall model.

[0210] In this embodiment, the search term can be text information composed of words, phrases, and / or sentences.

[0211] In some possible implementations, the search term can also be information determined after extracting or recognizing content in a picture, audio, or video.

[0212] For example, a user can input the two characters "tree leaves" as a search term through a search box of a search page.

[0213] Alternatively, the user can speak the word "tree leaves" into the search box through a search box voice input function, and the search engine can recognize the word "tree leaves" from the voice as a search term through a voice recognition function.

[0214] For another example, the user can upload a picture containing a tree leaf pattern to the search box, and the search engine can recognize the word "tree leaves" from the picture as a search term through an image recognition function.

[0215] In this embodiment, each single-path recall model can recall a plurality of relevant documents based on the search term, and perform relevance scoring on the matching degree (such as semantic matching degree) of the recalled relevant documents and the search term to obtain a corresponding relevance score. The documents recalled by the single-path recall model and the relevance scores of the recalled documents form the first recall result of the single-path recall model.

[0216] It can be understood that in some specific examples, each single-path recall model in the multi-path recall model can use different recall strategies or features to respectively recall a part of the documents when the multi-path recall model performs multi-path recall. For example, for a recall scene of shopping recommendation, each single-path recall model of the multi-path recall model can respectively use the strategies of "hot commodity", "user preference feature recall", "keyword recall", "commodity category recall", "collaborative filtering recall", and the like to perform single-path recall.

[0217] For example, referring back to the multi-path recall model shown in Figure 3 , after the user inputs the search term Q1 into the search engine, the search engine can perform multi-path recall on the search term Q1 to obtain a plurality of relevant documents, as shown in Figure 7As shown, the search engine performs discrete representation on the query Q1 through the first model in the multi-path recall model to obtain a vector q1, and then recalls a candidate set d1 based on the vector q1. In addition, the search engine also performs dense representation on the query Q1 through the second model in the multi-path recall model to obtain a vector q2, and then recalls a candidate set q2 based on the vector q2.

[0218] S602, fusing the first recall results of each single-path recall model by using the weight corresponding to each single-path recall model to obtain a second recall result.

[0219] In this embodiment, after obtaining the first recall results of the plurality of single-path recall models, the relevance scores of each document in the first recall results are multiplied by the respective corresponding weights, and then the results of multiplying each first recall result by the weight are added to obtain the fused recall result, i.e., the second recall result, which is output to a downstream ranking model.

[0220] For example, if the multi-path recall model includes M single-path recall models, the second recall result of the multi-path recall model can be represented as:

[0221]

[0222] In formula (8), x represents input data, i.e., a query. θ M is a model parameter of the Mth model, β M is a weight of the Mth model, f M (x, θ M ) is a first recall result of the Mth model, is a second recall result output by the multi-path recall model.

[0223] In this way, the recall effects of each single-path recall model in the multi-path recall model are complementary, and each single-path recall model can also balance the recall result of the single-path recall model through the weight obtained by training, which can solve the heterogeneity problem between the plurality of single-path recall models, thereby facilitating the improvement of the overall recall effect of the multi-path recall model. In addition, based on the complementary performance between the plurality of single-path recall models, different types of single-path recall models can be combined, such as the first model being a dense representation model and the second model being a sparse representation model, so as to realize an industrial-level joint optimization multi-path recall model framework in the search field, which has strong universality and can efficiently and quickly realize recall in multiple tasks and scenarios.

[0224] In addition, based on Figure 2 When the multi-path recall model shown in the above multi-path recall method is used for recall, offline testing is performed on a plurality of data sets, and compared with the recall model in a mature search engine, the following comparison results are obtained:

[0225] In the search experiment process on the open source real user data set, when the search experiment is carried out on the question answering (QA) search data set of Bing (such as MSMACRO, Trec19 and the like), compared with the recall effect of the mature retrieval models such as the deep contextualized term (DeepCT) model, the pre-training language model Doc2query, the model based on approximate nearest neighbor negative contrastive estimation (ANCE), the open domain question answering tool RocketQA, the pre-training language model of contrastive learning for sentence representation (CLEAR), the contextualized inverted list (COIL) retrieval model, and the sparterm retrieval model which simultaneously learns the document expansion and the term weight, the recall effect of the above-mentioned multi-path recall method based on the multi-path recall model in the embodiment of the application is superior to the above-mentioned mature retrieval models in the evaluation indexes such as Recall@1000 and MRR@10.

[0226] In the search experiment process on the simulated large-scale data set, the mixed text generated on the basis of the MSMACRO data set is used as the negative sample, and the generated large-scale data set is used for search experiment. Compared with the recall effect of the mature language models such as the BM25 model, the sentence-bert sentence representation model and the condenser sentence representation model, the error rate of the recall result after the above-mentioned multi-path recall based on the multi-path recall model trained through the above-mentioned negative sample resampling and the like is also superior to these mature language models.

[0227] In the search experiment process on the large search business data set, when the search experiment is carried out based on the large search query and the user clicked web page text data, compared with the recall effect of the sparterm retrieval model, the recall effect of the above-mentioned multi-path recall method based on the multi-path recall model in the embodiment of the application is superior to the sparterm retrieval model in the evaluation indexes such as the recall rate Recall@1000 and the mean reciprocal ranking (MRR@10).

[0228] Through search experiment comparison, the multi-path recall model trained by the embodiment of the application has a recall effect exceeding the state of the art (SOTA) model in a multi-path recall process on a public data set, and a relative improvement of 4.2% in MRR@10 on the MSMARCO data set and a relative improvement of 12% in MRR@10 on the TREC19 Doc data set. In addition, the multi-path recall model trained by the embodiment of the application is relatively stable in a large-scale search scenario in a multi-path recall process, and significantly exceeds the baseline model, with a relative improvement of 77.5% compared with the dense model. In addition, the multi-path recall model trained by the embodiment of the application has a relative improvement of more than 50% in MRR@10 offline effect compared with the sparterm retrieval model currently used by the large search.

[0229] In summary, the multi-path recall model trained by the embodiment of the application has strong versatility and high overall recall effect in a multi-path recall process.

[0230] Based on the multi-path recall model training method in the above embodiment, the embodiment of the application provides a multi-path recall model training device. Please refer to Figure 8 , Figure 8 is a structural schematic diagram of a multi-path recall model training device provided by the embodiment of the application.

[0231] As Figure 8 shown, the multi-path recall model training device 800 includes M single-path recall models, and M>2. The device 800 can include a first acquisition module 801 and a first processing module 802. The first acquisition module 801 can be configured to acquire a plurality of groups of training data, each group of training data including a target positive sample corresponding to a search term and a plurality of negative samples, each sample including a search term and a corresponding document. The first processing module 802 can be configured to obtain a relevance score of all samples in each group of training data by the M-1th single-path recall model, the relevance score representing a matching degree of the document in the corresponding sample and the search term. Then the first processing module 802 can be further configured to perform data fitting according to the relevance scores of each group of training data and a first loss function to determine a first expected recall result of the Mth model, the first loss function being used to determine a deviation between the M-1th model and a second expected recall result when recalling the corresponding training data. In addition, the first processing module 802 is further configured to determine a loss of the Mth single-path recall model according to the first expected recall result and a second loss function, the loss representing a deviation between the Mth single-path recall model and the first expected recall result when recalling the corresponding training data. Then the first processing module 802 is further configured to train the Mth single-path recall model with the minimum loss as the target.

[0232] In some embodiments, the first processing module 802 is further configured to determine a total loss of a preset neural network model according to each group of training data, the first weight, the second weight, and the proxy function, wherein the preset neural network model comprises the M single-path recall models, the total loss represents an error between output data of the neural network model and expected output data, the first weight comprises M-1 first weights, and the M-1 first weights are weights of the first single-path recall model to the (M-1)th single-path recall model, respectively, and the second weight is a weight of the Mth single-path recall model. Furthermore, the first processing module 802 is further configured to adjust a value of the second weight to train the neural network model in a direction in which the total loss is minimized, and determine a value of the second weight corresponding to the total loss when the training is stopped as a target weight of the Mth single-path recall model.

[0233] In some embodiments, the first processing module 802 is further configured to sequentially place values of the total loss in each round of training of the neural network model into a fixed-length sequence. Furthermore, when a variation rate of a difference between a maximum value and a minimum value in the fixed-length sequence is lower than a preset threshold within a preset time length, the training is stopped, and a value of the second weight corresponding to the total loss when the training is stopped is determined as the target weight.

[0234] In some embodiments, the first processing module 802 is further configured to obtain, by using a preset recall model, relevance scores of a plurality of documents recalled based on the search term, wherein the plurality of documents are from a preset data set, the data set is a set comprising a plurality of documents related to the search term, and part of the documents in the set are documents labeled as positive samples; then arrange the plurality of documents in descending order of the relevance scores, and determine documents ranked in the first T positions and not labeled as positive samples as first negative samples; and form a negative sample set by using a plurality of the first negative samples, so as to obtain the training data according to the negative sample set and the positive samples.

[0235] In some embodiments, the first processing module 802 is further configured to obtain, by using a first model, relevance scores of all documents recalled from a data set based on the search term, wherein the data set comprises a positive sample set and a negative sample set, and the negative sample set comprises a plurality of negative samples. Then the first processing module 802 determines a target sample according to the relevance scores of all the recalled documents, the target sample being a sample whose relevance score reaches a preset value and is labeled incorrectly. Furthermore, the first processing module 802 is further configured to obtain probability distribution data of all the negative samples in the negative sample set according to the relevance scores of the documents corresponding to the negative samples; and then select, according to the probability distribution data, negative samples distributed within a preset value range in the negative sample set as second negative samples, so that the second negative samples and the positive sample set form the training data, wherein the target sample is not included in the second negative samples.

[0236] It should be understood that the above-mentioned training device is used to execute the training method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned training method. The working process of the device can refer to the corresponding process in the above-mentioned training method and will not be repeated here.

[0237] Based on the multi-channel recall method in the above embodiment, the present application embodiment provides a multi-channel recall device. Figure 9 , Figure 9 2 is a structural diagram of a multi-channel recall device 900 provided in an embodiment of the present application.

[0238] like Figure 9 As shown, the multi-way recall device 900 may include a second acquisition module 901 and a second processing module 902. The second acquisition module 901 may be used to acquire a search term; the second processing module 902 may be used to input the search term into each single-way recall model in the multi-way recall model to obtain a first recall result of each single-way recall model; and the second processing module 902 may be used to fuse each first recall result using the fusion weight corresponding to each single-way recall model to obtain a second recall result.

[0239] It should be understood that the above-mentioned multi-channel recall device is used to execute the training method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned multi-channel recall method. The working process of the device can refer to the corresponding process in the above-mentioned multi-channel recall method, which will not be repeated here.

[0240] Based on the methods in the above embodiments, embodiments of the present application provide an electronic device. The electronic device may include: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is configured to execute the methods in the above embodiments.

[0241] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0242] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0243] Based on the method in the above embodiment, the present application embodiment also provides a chip. Figure 10 , Figure 10A structural schematic diagram of a chip is provided in the embodiments of the present application. As shown in the figure, Figure 10 The chip 1000 comprises one or more processors 1001 and interface circuits 1002. Optionally, the chip 1000 can further comprise a bus 1003. Wherein:

[0244] The processor 1001 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware form of the processor 1001 or the instruction in the software form. The processor 1001 described above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0245] The interface circuit 1002 can be used for sending or receiving data, instructions or information. The processor 1001 can process the data, instructions or other information received by the interface circuit 1002, and can send the processed information out through the interface circuit 1002.

[0246] Optionally, the chip 1000 further comprises a memory, which can include a read-only memory and a random access memory, and provides operation instructions and data for the processor. Part of the memory can also include a non-volatile random access memory (NVRAM).

[0247] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which can be stored in an operating system).

[0248] Optionally, the interface circuit 1002 can be used to output the execution result of the processor 1001.

[0249] It should be noted that the functions of the processor 1001 and the interface circuit 1002 respectively can be realized by hardware design, software design or combination of hardware and software, which is not limited here.

[0250] It should be understood that each step of the above method embodiment can be completed by the logic circuit in the hardware form of the processor or the instruction in the software form.

[0251] It can be understood that the size of the serial number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, each step in the above-mentioned embodiments can be selectively executed, partially executed or fully executed according to actual conditions, which is not limited here.

[0252] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0253] The method steps in the embodiments of the present application can be realized by hardware or by the processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0254] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0255] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

Claims

1. A multi-way recall model training method, characterized in that: The multi-way recall model includes M single-way recall models, M≥2, and the training method includes: Acquire multiple sets of training data, each set of training data includes a target positive sample and multiple negative samples corresponding to a search term, and each sample includes the search term and a corresponding document; Obtain the relevance scores of all samples in each set of training data through the M-1th single-way recall model, where the relevance scores represent the degree of match between the documents in the corresponding samples and the search terms; Performing data fitting based on the relevance scores of the training data of each group and a first loss function to determine a first expected recall result of the Mth single-way recall model, wherein the first loss function is used to determine a deviation between the M-1th single-way recall model and a second expected recall result when recalling the corresponding training data; Determining a loss of the Mth single-pass recall model based on the first expected recall result and the second loss function, where the loss represents a deviation between the Mth single-pass recall model and the first expected recall result when recalling corresponding training data; The M-th single-way recall model is trained with the goal of minimizing the loss.

2. The training method according to claim 1, characterized in that After training the M-th single-way recall model with the goal of minimizing the loss, the training method further includes: Determine the total loss of a preset neural network model based on each set of training data, the first weight, the second weight, and the proxy function, wherein the preset neural network model includes the M single-way recall models, the total loss represents the error between the output data of the neural network model and the expected output data, the first weight includes M-1, the M-1 first weights are respectively the weights of the 1st to M-1th single-way recall models, and the second weight is the weight of the Mth single-way recall model; Adjusting the value of the second weight to train the neural network model in a direction of minimizing the total loss; The value of the second weight corresponding to the total loss when training is stopped is determined as the target weight of the M-th single-way recall model.

3. The training method according to claim 2, characterized in that The step of determining the value of the second weight corresponding to the total loss when stopping training as the target weight of the M-th single-way recall model includes: Put the total loss value of each round of training of the neural network model into a fixed-length sequence in order; When the variable rate of the difference between the maximum value and the minimum value in the fixed-length sequence is lower than a preset threshold within a preset time length, the training is stopped; The value of the second weight corresponding to when the training is stopped is determined as the target weight.

4. The training method according to any one of claims 1 to 3, characterized in that: The obtaining of multiple sets of training data includes: Obtaining, by a preset recall model, relevance scores of a plurality of documents recalled based on the search term, wherein the plurality of documents are from a preset data set, the data set being a collection of documents related to the search term, and some of the documents in the collection being documents marked as positive samples; After arranging the multiple documents in descending order of their relevance scores, the documents that are ranked in the top T positions and are not marked as positive samples are determined as the first negative samples; A negative sample set is formed by using a plurality of the first negative samples, so as to obtain the training data according to the negative sample set and the positive samples.

5. The training method according to any one of claims 1 to 3, characterized in that: The step of obtaining multiple sets of training data further includes: Obtaining, using the first model, relevance scores of all documents recalled from a data set based on the search term, wherein the data set includes a positive sample set and a negative sample set, wherein the negative sample set includes a plurality of negative samples; According to the relevance scores of all recalled documents, target samples are determined. The target samples are samples whose relevance scores reach the preset value and are marked incorrectly. Obtaining probability distribution data of all negative samples according to the relevance scores of the documents corresponding to all negative samples in the negative sample set; According to the probability distribution data, negative samples in the negative sample set distributed within a preset value range are selected as second negative samples, so that the second negative samples and the positive sample set form the training data, wherein the second negative samples do not include the target sample.

6. A multi-channel recall method, characterized in that: The method is implemented by the multi-channel recall model according to any one of claims 1 to 5, wherein the multi-channel recall model includes M single-channel recall models, where M is greater than or equal to 2, and the multi-channel recall method includes: Inputting the search terms into each of the single-way recall models in the multi-way recall model to obtain a first recall result of each of the single-way recall models; The first recall results are fused using the weights corresponding to the single-path recall models to obtain a second recall result.

7. A multi-channel recall model training device, characterized in that: The multi-way recall model includes M single-way recall models, M≥2, and the training device includes: A first acquisition model is used to acquire multiple sets of training data, each set of training data includes a target positive sample and multiple negative samples corresponding to a search term, and each sample includes the search term and a corresponding document; A first processing module is configured to obtain, through the M-1th single-way recall model, a relevance score of all samples in each set of training data, wherein the relevance score represents a degree of matching between the document in the corresponding sample and the search term; The first processing module is further configured to perform data fitting based on the relevance scores of the training data groups and a first loss function to determine a first expected recall result of the Mth single-way recall model, wherein the first loss function is configured to determine a deviation between the M-1th single-way recall model and a second expected recall result when recalling the corresponding training data; The first processing module is further configured to determine a loss of the Mth single-pass recall model based on the first expected recall result and a second loss function, wherein the loss represents a deviation between the Mth single-pass recall model and the first expected recall result when recalling corresponding training data; The first processing module is further configured to train the Mth single-way recall model with the goal of minimizing the loss.

8. The device according to claim 7, characterized in that The first processing module is further configured to determine a total loss of a preset neural network model based on each set of training data, the first weight, the second weight, and the proxy function, wherein the preset neural network model includes the M single-way recall models, the total loss represents an error between the output data of the neural network model and the expected output data, the first weight includes M-1, the M-1 first weights are respectively the weights of the 1st to M-1th single-way recall models, and the second weight is the weight of the Mth single-way recall model; The first processing module is further configured to adjust the value of the second weight so as to train the neural network model in a direction in which the total loss is minimized; The first processing module is further configured to determine the value of the second weight corresponding to the total loss when training is stopped as the target weight of the Mth single-way recall model.

9. The device according to claim 7 or 8, characterized in that The first processing module is further configured to sequentially place the total loss values ​​during each round of training of the neural network model into a fixed-length sequence; The first processing module is also used to stop training when the variable rate of the difference between the maximum value and the minimum value in the fixed-length sequence is lower than a preset threshold within a preset time length, and determine the value of the second weight corresponding to the time when the training is stopped as the target weight.

10. The device according to any one of claims 7 to 9, characterized in that: The first processing module is further configured to obtain, by using a preset recall model, relevance scores of a plurality of documents recalled based on the search term, wherein the plurality of documents are from a preset data set, the data set being a collection of documents related to the search term, and some of the documents in the collection being documents marked as positive samples; The first processing module is further configured to arrange the plurality of documents in descending order of their relevance scores, and determine the documents that are ranked in the top T positions and are not marked as positive samples as first negative samples; The first processing module is further configured to use the plurality of the first negative samples to form a negative sample set, so as to obtain the training data according to the negative sample set and the positive samples.

11. The device according to any one of claims 7 to 9, characterized in that: The first processing module is further configured to obtain, using the first model, relevance scores of all documents recalled from a data set based on the search term, wherein the data set includes a positive sample set and a negative sample set, wherein the negative sample set includes a plurality of negative samples; The first processing module is further configured to determine target samples based on the relevance scores of all recalled documents, where the target samples are samples whose relevance scores reach a preset value and are marked as errors; The first processing module is further configured to obtain probability distribution data of all negative samples in the negative sample set based on the relevance scores of the documents corresponding to all negative samples; The first processing module is further used to select, based on the probability distribution data, negative samples in the negative sample set that are distributed within a preset value range as second negative samples, so that the second negative samples and the positive sample set form the training data, wherein the second negative samples do not include the target sample.

12. A multi-channel recall device, characterized in that: The device includes a second acquisition model and a second processing module, The second acquisition module is used to obtain the search term; a second processing module, configured to input the search terms into each single-way recall model in the multi-way recall model according to any one of claims 7 to 11, respectively, to obtain a first recall result of each single-way recall model; The second processing module is further configured to fuse the first recall results using the weights corresponding to the single-path recall models to obtain a second recall result.

13. An electronic device, characterized in that: include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; When the program stored in the memory is executed, the processor is used to execute the training method according to any one of claims 1 to 5 or the multi-way recall method according to claim 6.

14. A computer-readable storage medium storing a computer program, wherein when the computer program runs on a processor, the processor executes the training method according to any one of claims 1 to 5 or the multi-way recall method according to claim 6.

15. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the training method according to any one of claims 1 to 5 or the multi-way recall method according to claim 6.

16. A chip, characterized in that: comprising at least one processor and an interface; The at least one processor obtains program instructions or data through the interface; The at least one processor is used to execute the program line instructions to implement the training method according to any one of claims 1 to 5 or the multi-way recall method according to claim 6.

Images