Generative retrieval model training method, information retrieval method and device

Abstract

One or more embodiments of the invention provide a training method of a generative retrieval model, and an information retrieval method and device. The training method adopts a two-stage training strategy. Wherein the first stage can be called a document identifier generation stage, and in the stage, first query entries and first standard document identifiers which are in one-to-one correspondence are utilized to train a to-be-trained generative retrieval model to learn the relationship between query and document identifiers. The second stage can be called a document ranking stage, and in the stage, the relationship between the model trained in the first stage is trained by utilizing second query entries, second positive sample documents and negative sample documents which are in one-to-one correspondence through a comparative learning method to learn the query and the documents.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of information retrieval technology, and in particular, to a training method for a generative retrieval model, an information retrieval method, and a device. Background Art

[0002] The search advertising system is an advertising display and delivery system based on the search engine platform. The search advertising system can recall multiple related advertisements based on the search query entered by the user, and sort the multiple related advertisements according to certain rules. Finally, the search advertising system displays the sorted advertisement list to the user for selection and click.

[0003] Among them, the accuracy of the recall result is an important factor affecting the user experience. Therefore, it is necessary to provide a method to make the recall result more accurate. Summary of the invention

[0004] In order to improve the accuracy of recall results, one or more embodiments of the present specification provide a training method for a generative retrieval model, an information retrieval method, and an apparatus.

[0005] In a first aspect, one or more embodiments of the present specification provide a method for training a generative retrieval model, the method comprising: obtaining first training data, the first training data comprising one-to-one corresponding first query terms and first positive sample documents; generating first standard document identifiers corresponding to each first positive sample document; using the one-to-one corresponding first query terms and first standard document identifiers, training a generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers, thereby obtaining a first generative retrieval model; obtaining second training data, the second training data comprising one-to-one corresponding second query terms, second positive sample documents and negative sample documents; using a comparative learning method, using the second training data to train the first generative retrieval model to learn the correlation between the semantic representation of the query terms and the semantic representation of the document, thereby obtaining a second generative retrieval model.

[0006] In one possible implementation, a generative retrieval model to be trained is used to learn the correlation between the query terms and the document identifiers using one-to-one correspondence between the first query terms and the first standard document identifiers to obtain a first generative retrieval model, including: sampling the corresponding first positive document fragments from each first positive sample document respectively; inputting the first query term and the first positive document fragment into the generative retrieval model to be trained respectively; outputting the first predicted document identifier corresponding to the first query term and the second predicted document identifier corresponding to the first positive document fragment through the generative retrieval model to be trained; determining a first generation loss based on the first standard document identifier, the first predicted document identifier and the second predicted document identifier; and training the generative retrieval model to be trained to convergence based on the first generation loss to obtain a first generative retrieval model.

[0007] In one possible implementation, the negative sample document includes a first negative sample document and a second negative sample document; wherein the identifier prefix of the first negative sample document is partially or completely the same as the identifier prefix of the second positive sample document; and the identifier prefix of the second negative sample document is different from the identifier prefix of the second positive sample document.

[0008] In one possible implementation, the method also includes: generating a second standard document identifier corresponding to the second positive sample document; sampling a first preset number of documents from the document library as first negative sample documents in order from most to least number of documents having the same identifier prefix as the second standard document identifier.

[0009] In one possible implementation, the method also includes: inputting a second query term into a first generative retrieval model; outputting a plurality of candidate document identifiers related to the second query term through the first generative retrieval model; generating a second standard document identifier corresponding to a second positive sample document; and sampling a second preset number of documents from documents corresponding to candidate document identifiers that are different from the second standard document identifier as second negative sample documents.

[0010] In one possible implementation, the first generative retrieval model includes an encoder; the first generative retrieval model is trained using the second training data through a contrastive learning method to learn the correlation between the semantic representation of the query term and the semantic representation of the document to obtain the second generative retrieval model, including: respectively inputting the second query term, the second positive sample document, the first negative sample document, and the second negative sample document into the first generative retrieval model; outputting the query semantic representation corresponding to the second query term, the positive sample semantic representation corresponding to the second positive sample document, the first negative sample semantic representation corresponding to the first negative sample document, and the second negative sample semantic representation corresponding to the second negative sample document through the encoder; determining the positive sample semantic score corresponding to the positive sample semantic representation, the first negative sample semantic score corresponding to the first negative sample semantic representation, and the second negative sample semantic score corresponding to the second negative sample semantic representation based on the query semantic representation; determining the contrast loss based on the positive sample semantic score, the first negative sample semantic score, and the second negative sample semantic score; training the first generative retrieval model to convergence based on the contrastive loss to obtain the second generative retrieval model.

[0011] In one possible implementation, the first generative retrieval model also includes a decoder, and the first generative retrieval model is trained to convergence based on the contrast loss to obtain a second generative retrieval model, including: sampling corresponding second positive document fragments from each second positive sample document; generating third standard document identifiers corresponding to the second positive sample document, the first negative sample document, and the second negative sample document respectively; inputting the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document into the first generative retrieval model respectively; outputting the third predicted document identifiers corresponding to the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document respectively through the decoder; determining the second generation loss based on the third standard document identifier and the third predicted document identifier corresponding to the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document respectively; and training the first generative retrieval model to convergence based on the second generation loss and the contrast loss to obtain the second generative retrieval model.

[0012] In a second aspect, one or more embodiments of the present specification also provide an information retrieval method, applied to an electronic device, the method comprising: receiving an input query term; generating the first k document identifiers corresponding to the query term and the probability values ​​corresponding to the first k document identifiers respectively using a second generative retrieval model trained by any method of the first aspect; wherein k is a positive integer greater than or equal to 1; determining the candidate documents corresponding to the first k document identifiers respectively; determining the semantic scores corresponding to the candidate documents respectively based on the query semantic representation of the query term and the document semantic representations of the candidate documents; wherein the query semantic representation and the document semantic representations are obtained by the second generative retrieval model; determining the relevance scores corresponding to the candidate documents respectively based on the probability values ​​and semantic scores corresponding to the candidate documents respectively; sorting the candidate documents based on the relevance scores, and using the sorted candidate documents as retrieval results.

[0013] In a possible implementation, the query term is input information received through a search box on an application page of an electronic device, and the candidate document is an advertisement.

[0014] In a third aspect, one or more embodiments of this specification further provide a training device for a generative retrieval model, the device comprising:

[0015] A first acquisition module, used to acquire first training data, the first training data including a one-to-one correspondence between a first query term and a first positive sample document;

[0016] A first generating module, used to generate first standard document identifiers corresponding to each first positive sample document;

[0017] A first training module is used to train the generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers using the one-to-one correspondence between the first query terms and the first standard document identifiers, so as to obtain a first generative retrieval model;

[0018] A second acquisition module, used to acquire second training data, the second training data including a one-to-one correspondence between a second query term, a second positive sample document, and a negative sample document;

[0019] The second training module is used to train the first generative retrieval model by using the second training data through a comparative learning method to learn the correlation between the semantic representation of the query term and the semantic representation of the document, so as to obtain a second generative retrieval model.

[0020] In one possible implementation, the first training module is specifically used to: sample the corresponding first positive document fragments from each first positive sample document; input the first query term and the first positive document fragment into the generative retrieval model to be trained; output the first predicted document identifier corresponding to the first query term and the second predicted document identifier corresponding to the first positive document fragment through the generative retrieval model to be trained; determine the first generation loss based on the first standard document identifier, the first predicted document identifier and the second predicted document identifier; train the generative retrieval model to be trained until convergence based on the first generation loss to obtain a first generative retrieval model.

[0021] In one possible implementation, the negative sample document includes a first negative sample document and a second negative sample document; wherein the identifier prefix of the first negative sample document is partially or completely the same as the identifier prefix of the second positive sample document; and the identifier prefix of the second negative sample document is different from the identifier prefix of the second positive sample document.

[0022] In one possible implementation, the device also includes: a first negative sample document sampling module, used to generate a second standard document identifier corresponding to the second positive sample document; and sample a first preset number of documents from the document library as first negative sample documents in order from most to least number of documents having the same identifier prefix as the second standard document identifier.

[0023] In one possible implementation, the device also includes: a second negative sample document sampling module, which is used to input the second query term into the first generative retrieval model; output multiple alternative document identifiers related to the second query term through the first generative retrieval model; generate a second standard document identifier corresponding to the second positive sample document; and sample a second preset number of documents from the documents corresponding to the alternative document identifiers that are different from the second standard document identifier as second negative sample documents.

[0024] In one possible implementation, the first generative retrieval model includes an encoder; a second training module is specifically used to: input the second query term, the second positive sample document, the first negative sample document, and the second negative sample document into the first generative retrieval model respectively; output the query semantic representation corresponding to the second query term, the positive sample semantic representation corresponding to the second positive sample document, the first negative sample semantic representation corresponding to the first negative sample document, and the second negative sample semantic representation corresponding to the second negative sample document through the encoder; determine the positive sample semantic score corresponding to the positive sample semantic representation, the first negative sample semantic score corresponding to the first negative sample semantic representation, and the second negative sample semantic score corresponding to the second negative sample semantic representation based on the query semantic representation; determine the contrast loss based on the positive sample semantic score, the first negative sample semantic score, and the second negative sample semantic score; train the first generative retrieval model to convergence based on the contrast loss to obtain the second generative retrieval model.

[0025] In one possible implementation, the first generative retrieval model also includes a decoder and a second training module, which are specifically used to: sample corresponding second positive document fragments from each second positive sample document; generate third standard document identifiers corresponding to the second positive sample document, the first negative sample document, and the second negative sample document; input the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document into the first generative retrieval model respectively; output the third predicted document identifiers corresponding to the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document respectively through the decoder; determine the second generation loss based on the third standard document identifiers and the third predicted document identifiers corresponding to the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document respectively; and train the first generative retrieval model to convergence based on the second generation loss and the contrast loss to obtain the second generative retrieval model.

[0026] In a fourth aspect, one or more embodiments of this specification further provide an information retrieval device, applied to an electronic device, the device comprising:

[0027] A receiving module, used for receiving an input query term;

[0028] A second generation module, for generating the first k document identifiers corresponding to the query terms and the probability values ​​corresponding to the first k document identifiers respectively, using the second generative retrieval model trained by any one of the methods of the first aspect; wherein k is a positive integer greater than or equal to 1;

[0029] A first determination module is used to determine candidate documents corresponding to the first k document identifiers respectively;

[0030] A second determination module is used to determine the semantic score corresponding to each candidate document based on the query semantic representation of the query term and the document semantic representation of each candidate document; wherein the query semantic representation and the document semantic representation are obtained through a second generative retrieval model;

[0031] A third determination module is used to determine the relevance score corresponding to each candidate document based on the probability value and semantic score corresponding to each candidate document;

[0032] The sorting module is used to sort the candidate documents based on the relevance scores and use the sorted candidate documents as the retrieval results.

[0033] In a possible implementation, the query term is input information received through a search box on an application page of an electronic device, and the candidate document is an advertisement.

[0034] In a fifth aspect, one or more embodiments of the present specification further provide an electronic device comprising a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method of the first aspect or the second aspect mentioned above is implemented.

[0035] In a sixth aspect, one or more embodiments of the present specification further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the method of the first aspect or the second aspect described above is implemented.

[0036] In summary, one or more embodiments of the present specification respectively train the model to learn the correlation between query terms and document identifiers, and the correlation between the semantic representation of the query terms and the semantic representation of the document through two-stage training. In addition, the first negative sample document introduced during contrastive learning can be used to capture the semantic representation of documents with the same identifier prefix, and the second negative sample document introduced can be used to capture the semantic representation of documents with different identifier prefixes. Based on the above two types of negative sample documents, two corresponding contrast losses are applied to maximize the semantic score between the query and the positive document to achieve their similar representation.

[0037] In addition, the retrieval results obtained by sorting based on the relevance score in the information retrieval method take into account both the association between the query terms and the document identifiers and the association between the query terms and the documents, making the retrieval results more accurate. Moreover, when the same document identifier corresponds to multiple candidate documents, the semantic score can be used to solve the problem of inaccurate candidate document sorting due to document identifier conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of one or more embodiments of the present specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of one or more embodiments of the present specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A flowchart of a training method for a generative retrieval model provided for one or more embodiments of this specification;

[0040] Figure 2 A flowchart of another method for training a generative retrieval model provided for one or more embodiments of this specification;

[0041] Figure 3A flowchart of a training method for a document identifier generation phase provided for one or more embodiments of this specification;

[0042] Figure 4 A flowchart of a training method for a document ranking stage provided for one or more embodiments of this specification;

[0043] Figure 5 A flowchart of an information retrieval method provided for one or more embodiments of this specification;

[0044] Figure 6 A flowchart of another information retrieval method provided for one or more embodiments of this specification;

[0045] Figure 7 A structural block diagram of a training device for a generative retrieval model provided in one or more embodiments of this specification;

[0046] Figure 8 A structural block diagram of an information retrieval device provided for one or more embodiments of this specification;

[0047] Fig. 9 A structural block diagram of an electronic device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0048] One or more embodiments of the present invention are further described in detail below through the accompanying drawings and examples. Through these descriptions, the features and advantages of one or more embodiments of the present invention will become clearer and more specific.

[0049] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0050] In addition, the technical features involved in different implementations of one or more embodiments of this specification described below can be combined with each other as long as they do not conflict with each other.

[0051] To facilitate understanding, the application scenarios of the technical solutions provided by one or more embodiments of this specification are first described below.

[0052] The search advertising system is an advertising display and delivery system based on a search engine platform. The search advertising system can recall multiple related advertisements (such as mini-program advertisements) based on the search query entered by the user, and sort the multiple related advertisements according to certain rules. Finally, the search advertising system displays the sorted advertisement list to the user for selection and click. Among them, mini-program advertisements refer to advertisements displayed on the mini-program page.

[0053] This recall mode can be called a one-stage recall mode, which realizes the recall process from user search directly to relevant advertisements, and does not rely on the specific bidwords of advertisers. The one-stage recall mode significantly simplifies the recall process and improves recall efficiency and user experience.

[0054] It can be seen that the one-stage recall mode in the search advertising system can be understood as an information retrieval scheme, in which the advertisement as the retrieval object can be regarded as a document. For example, the one-stage recall mode can be a sparse retrieval scheme or a dense retrieval scheme based on the "index-recall-rearrangement" retrieval architecture. However, the sparse retrieval scheme cannot capture the rich semantic relationship between the query and the document, and relies on vocabulary overlap, and cannot match documents with a small amount of word overlap but semantically related. Dense retrieval schemes generally lack interaction due to their computational overhead and cannot achieve end-to-end modeling. At the same time, the extracted representation is greatly affected by the length of the text information.

[0055] One or more embodiments of the present specification provide a generative retrieval model, and use the generative retrieval model to retrieve recall results. The generative retrieval model is a generative retrieval architecture based on a large language model.

[0056] In the information retrieval scheme using generative retrieval models, a document identifier (docid) can be assigned to each document. Then, a ranked list of document identifiers is generated for a given query through the generative retrieval model end-to-end.

[0057] In one possible approach, a generative retrieval model is trained using one-to-one correspondence between query terms and document identifiers as training data. However, since the length of a document identifier is usually much smaller than the length of a document, there may be a certain degree of information loss in the document identifier compared to the complete document. Therefore, this training method mainly focuses on learning the relationship between the query and the document identifier, but cannot capture the complete semantic relationship between the query and the document.

[0058] To this end, one or more embodiments of this specification provide a training method for a generative retrieval model, such as Figure 1As shown, the training method adopts a two-stage training strategy. Among them, the first stage can be called the document identifier generation stage, which is used to train the generative retrieval model to be trained to learn the relationship between the query and the document identifier. The second stage can be called the document ranking stage, which is used to train the model trained in the first stage to learn the relationship between the query and the document. For the convenience of description, the model obtained by the first stage training can be called the first generative retrieval model, and the model obtained by the second stage training can be called the second generative retrieval model. It should be understood that the model structure of the first generative retrieval model and the second generative retrieval model are the same, but the model parameters are different.

[0059] In this way, the above two-stage training method can learn the generation probability at the identifier level and learn the semantic representation at the document level through the direct interaction between the query and the document to capture the complete semantic relationship between the query and the document.

[0060] It should be noted that one or more embodiments of the present specification do not limit the model architecture of the generative retrieval model. For example, the generative retrieval model may include an encoder and a decoder.

[0061] The following is an illustrative explanation of the training method in the document identifier generation phase.

[0062] like Figure 2 As shown, the document identifier generation phase in the training method of the generative retrieval model provided by one or more embodiments of this specification may include the following steps:

[0063] Step S102: Acquire first training data, where the first training data includes a one-to-one correspondence between a first query term and a first positive sample document.

[0064] Step S104: Generate first standard document identifiers corresponding to each first positive sample document.

[0065] Step S106, using the one-to-one correspondence between the first query terms and the first standard document identifier, the generative retrieval model to be trained learns the correlation between the query terms and the document identifier, and obtains a first generative retrieval model.

[0066] In step S102, the first positive sample document refers to the document most relevant to the first query term.

[0067] One or more embodiments of this specification do not limit the specific form of the first standard document identifier in step S104. For example, the first standard document identifier may be a keyword-based identifier, or a vocabulary-based or numerical-based identifier.

[0068] For example, a keyword extractor (e.g., pre-trained KeyBert) can be used to extract the first N keywords of document d and sort them in descending order of importance to obtain a keyword sequence. After encoding the keyword sequence using a word segmenter, the first n tokens are selected as document identifiers. In this way, the first standard document identifier corresponding to the document d can be obtained. According to the above method, the first standard document identifier corresponding to each first positive sample document can be generated.

[0069] It should be noted that the same document identifier may be generated for similar documents, and therefore, the same first standard document identifier may correspond to a group of documents.

[0070] After generating the first standard document identifiers corresponding to the first positive sample documents, a one-to-one correspondence between the first query terms and the first standard document identifiers can be obtained.

[0071] In step S106, the generative retrieval model to be trained may be trained with the first query term as input and the document identifier as output.

[0072] Exemplarily, the first query term may be input into the generative retrieval model to be trained. Then, the first predicted document identifier corresponding to the first query term is outputted by the generative retrieval model to be trained. Afterwards, a first generation loss is determined based on the first standard document identifier and the first predicted document identifier, and the generative retrieval model to be trained is trained to convergence based on the first generation loss to obtain a first generative retrieval model.

[0073] In this way, the first generative retrieval model trained in the first stage has the ability to generate document identifiers.

[0074] In one or more embodiments of the present specification, in the first stage, not only the relationship between query terms and document identifiers can be learned, but also the relationship between documents and document identifiers can be preliminarily learned.

[0075] In one possible implementation, Figure 3 As shown, the first positive sample document in the first training data can be input into the keyword extractor to obtain the first standard document identifier. And, the first positive document fragment corresponding to each first positive sample document in the first training data is sampled. In this way, the first positive document fragment and the first standard document identifier corresponding to the first query term can be obtained.

[0076] Afterwards, the first query term in the first training data and the first positive document fragment corresponding to the first query term are respectively input into the generative retrieval model to be trained, and the first predicted document identifier corresponding to the first query term and the second predicted document identifier corresponding to the first positive document fragment are output through the generative retrieval model to be trained. Then, the first generation loss can be determined based on the first standard document identifier, the first predicted document identifier and the second predicted document identifier. Finally, based on the first generation loss, the generative retrieval model to be trained is trained to convergence to obtain a first generative retrieval model.

[0077] Exemplarily, the first generation loss may be calculated using a cross entropy loss function. For example, the first generation loss may be calculated using the following formula (1):

[0078]

[0079] Where L1 represents the first generation loss value, n represents the number of samples in the first training data set, and t represents the index of the training sample. represents the probability distribution of document identifiers generated by the first query term input, Represents the probability distribution of document identifiers generated from the first positive document fragment of the input.

[0080] The model can be trained to learn the relationship between the query and the document identifier by generating a loss between the first standard document identifier and the first predicted document identifier. The model can be trained to learn the relationship between the query and the document by generating a loss between the first standard document identifier and the second predicted document identifier.

[0081] It should be noted that if the first generation loss value is less than or equal to the generation loss threshold, the training of the generative retrieval model to be trained can be stopped to obtain the first generative retrieval model. If the first generation loss value is greater than the generation loss threshold, the next round of training will continue until the first generation loss value is less than or equal to the generation loss threshold.

[0082] In this way, compared to training the relationship between the query and the document identifier separately, by taking the randomly sampled first positive document fragment as the input of the model, the model can learn the relationship between the document and the document identifier more comprehensively, thereby improving the robustness of the model in the task of generating document identifiers.

[0083] After the first generative retrieval model is obtained through the training in the document identifier generation phase, the trained first generative retrieval model may continue to be trained in the document ranking phase.

[0084] The following is an exemplary description of the training method for the document ranking stage provided in one or more embodiments of this specification.

[0085] Please continue to see Figure 2 In the training method of the generative retrieval model provided in one or more embodiments of this specification, the document ranking stage may include the following steps:

[0086] Step S108, obtaining second training data, the second training data including a one-to-one correspondence between a second query term, a second positive sample document, and a negative sample document.

[0087] Step S110, using a comparative learning method, the first generative retrieval model is trained using the second training data to learn the correlation between the semantic representation of the query term and the semantic representation of the document, thereby obtaining a second generative retrieval model.

[0088] In step S108, the second query term in the second training data may be the same as or different from the first query term in the first training data, and the second positive sample document in the second training data may be the same as or different from the first positive sample document in the first training data, and one or more embodiments of the present specification do not limit this.

[0089] In one or more embodiments of the present specification, the negative sample document may include a first negative sample document and a second negative sample document. The identifier prefix of the first negative sample document is partially or completely the same as the identifier prefix of the second positive sample document; the identifier prefix of the second negative sample document is different from the identifier prefix of the second positive sample document.

[0090] In a possible implementation, the first negative sample document can be obtained by the following method: first generate a second standard document identifier corresponding to the second positive sample document, and then sample a first preset number of documents from the document library in descending order of the number of documents having the same identifier prefix as the second standard document identifier as the first negative sample document.

[0091] Exemplarily, the second standard document identifier generated based on the second positive sample document d includes n tags, such as the second standard document identifier is Among them, n markers are sorted in order That is, the identifier prefix. After that, documents with the same identifier prefix as the first n identifier prefixes, the first n-1 identifier prefixes, the first n-2 identifier prefixes, and so on, the first nm identifier prefixes are searched from the document library in sequence until the first preset number of documents are found as the first negative sample documents. Where n is an integer greater than 1, and m is an integer greater than 0 and less than n.

[0092] It should be noted that the second positive sample document d itself is excluded from the documents having the same identifier prefix as the first n identifier prefixes, that is, the obtained first negative sample document does not include the corresponding second positive sample document d itself.

[0093] It should also be noted that the identifier prefix that is the same as the first n identifier prefixes refers to the identifier prefixes that have the same first n tags and order. Similarly, the identifier prefix that is the same as the first n-1 identifier prefixes refers to the identifier prefixes that have the same first n-1 tags and order. This is not listed here.

[0094] In this way, since document identifiers with the same identifier prefix often have similar document information, the identifier prefix is ​​used as a guide to gradually reduce the length of the identifier prefix and try to obtain documents with more identical identifier prefixes as the second standard document identifier from the document library as the first negative sample documents. This can help the model better capture the semantic representation of documents sharing the same identifier prefix.

[0095] In a possible implementation, the second negative sample document can be obtained in the following manner: input the second query term into the first generative retrieval model, and output the first k candidate document identifiers related to the second query term through the first generative retrieval model, where k is a positive integer greater than or equal to 1. Generate a second standard document identifier corresponding to the second positive sample document. Then, sample a second preset number of documents from the documents corresponding to the candidate document identifiers that are different from the second standard document identifier as the second negative sample documents.

[0096] Exemplarily, since the first generative retrieval model obtained through the first stage of training already has the ability to generate document identifiers, the first generative retrieval model can be used to generate a highly ranked document identifier for the input second query term. For example, the highly ranked document identifier includes k candidate document identifiers. Among them, the k candidate document identifiers are document identifiers corresponding to the k documents most relevant to the second query term. If the k candidate document identifiers include a document identifier that is the same as the second standard document identifier, the candidate document identifier that is the same as the second standard document identifier is removed from the k candidate document identifiers, and then the second negative sample document is selected from the documents corresponding to the remaining candidate document identifiers. If the k candidate document identifiers do not include a document identifier that is the same as the second standard document identifier, the second negative sample document is selected from the documents corresponding to the k candidate document identifiers.

[0097] In this way, by using the document corresponding to the document identifier different from the second standard document identifier in the retrieval results as the second negative sample document, it can help the model better understand the representation of documents with different identifier prefixes.

[0098] After obtaining the second positive sample documents, the first negative sample documents, and the second negative sample documents corresponding one by one to each second query term, the second query term, the second positive sample document, the first negative sample document, and the second negative sample document can be used as training data to train the first generative retrieval model through a comparative learning method.

[0099] In one possible implementation, Figure 4 As shown, the first generative retrieval model is trained by the contrastive learning method, which can be implemented in the following manner: the second query term, the second positive sample document, the first negative sample document and the second negative sample document are respectively input into the first generative retrieval model, and the query semantic representation corresponding to the second query term, the positive sample semantic representation corresponding to the second positive sample document, the first negative sample semantic representation corresponding to the first negative sample document and the second negative sample semantic representation corresponding to the second negative sample document are output through the encoder of the first generative retrieval model. Then, the positive sample semantic score corresponding to the positive sample semantic representation, the first negative sample semantic score corresponding to the first negative sample semantic representation and the second negative sample semantic score corresponding to the second negative sample semantic representation are determined based on the query semantic representation. Afterwards, the contrast loss is determined based on the positive sample semantic score, the first negative sample semantic score and the second negative sample semantic score, and the first generative retrieval model is trained to convergence based on the contrast loss to obtain the second generative retrieval model.

[0100] Semantic representation refers to representing the semantic information in a query term or document in the form of a vector. These vectors can reflect the subject, meaning, and contextual information of the query term or document.

[0101] Exemplarily, the semantic score between the query and the document can be calculated using the following formula (2):

[0102]

[0103] Among them, h q represents the query semantic representation, Represents the semantic representation of a document, for example, It can be a positive sample semantic representation, a first negative sample semantic representation, or a second negative sample semantic representation. s(q,d) represents the semantic score between the query and the document.

[0104] For example, When it is the semantic representation of the positive sample, s(q,d) is the semantic score of the positive sample; When it is the semantic representation of the first negative sample, s(q,d) is the semantic score of the first negative sample; When it is the semantic representation of the second negative sample, s(q,d) is the semantic score of the second negative sample.

[0105] The contrast loss may include the contrast loss L between the positive sample semantic score and the first negative sample semantic score. ci , and the contrast loss L between the semantic score of the positive sample and the semantic score of the second negative sample cq For example, the contrast loss can be the contrast loss L ci With contrast loss L cq The sum.

[0106] For example, the contrast loss L can be calculated using the following formula (3): ci , use the following formula (4) to calculate the contrast loss L cq :

[0107]

[0108] In formula (3), Nz represents the first negative sample document set, s(q,d + ) represents the semantic score of the positive sample, s(q,d - ) represents the semantic score of the first negative sample. In formula (4), Nq represents the second negative sample document set, s(q,d + ) represents the semantic score of the positive sample, s(q,d - ) represents the semantic score of the second negative sample. τ is the temperature parameter.

[0109] In this way, the second stage of training can learn the semantic representation of documents through direct interaction between queries and documents, and optimize the document representation through comparative learning of negative sample documents extracted from different perspectives.

[0110] It should be noted that one or more embodiments of this specification only use the training data including the first negative sample document and the second negative sample document to train the first generative retrieval model as an example, and do not limit the training data. For example, the training data of the second stage training may also include any one of the first negative sample document and the second negative sample document.

[0111] In order to prevent the second stage from using contrastive learning as a target, which results in a decrease in the generation capability of the document identifier of the second generative retrieval model, one or more embodiments of the present specification may reintroduce generation loss in the second stage, and jointly supervise model training through generation loss and contrastive loss, thereby ensuring the generation capability of the document identifier of the second generative retrieval model.

[0112] In one possible implementation, Figure 4As shown, by jointly supervising the model training through the generative loss and the contrastive loss, it can be implemented in the following way: the keyword extractor can be used to generate the third standard document identifiers corresponding to the second positive sample document, the first negative sample document and the second negative sample document respectively. For example, the standard document identifier corresponding to the second positive sample document is The standard document identifier corresponding to the first negative sample document is The standard document identifier corresponding to the second negative sample document is

[0113] A second positive document segment can also be sampled from the second positive sample document, and the second positive document segment is also input into the first generative retrieval model ( Figure 4 Not shown, see Figure 3 ). Then, the decoder of the first generative retrieval model outputs the third predicted document identifier z corresponding to the second query term d＇ , the third predicted document identifier corresponding to the second positive document segment The third predicted document identifier corresponding to the first negative sample document And the third predicted document identifier corresponding to the second negative sample document

[0114] Finally, the second generation loss L2 may be determined based on the third standard document identifier and the third predicted document identifier corresponding to the second query term, the second positive document segment, the first negative sample document, and the second negative sample document, respectively.

[0115] For example, we can use the cross entropy loss function to calculate the standard document identifier and the predicted document identifier z d＇ The generation loss between the standard document identifier and predict document identifiers The generation loss between the standard document identifier and predict document identifiers The generation loss between the standard document identifier and predict document identifiers Afterwards, the sum of the above four generation losses is taken as the second generation loss L2.

[0116] In this way, during the second stage of training, the second generation loss L2 and the contrast loss can be used together as the loss of the first generative retrieval model to train it.

[0117] For example, the loss L corresponding to the second stage training is L = L ci +L cq +λL2. Among them, λ is a hyperparameter that controls the importance of the second generation loss, which is used to balance the generation goal and the contrastive learning goal.

[0118] In this way, during the second stage training process, the second generation loss and the contrast loss are used to jointly supervise the model training, so as to ensure that the document identifier generation ability of the second generative retrieval model will not decrease.

[0119] One or more embodiments of the present specification use two-stage training to train the model to learn the correlation between query terms and document identifiers, and the correlation between the semantic representation of the query terms and the semantic representation of the document. In addition, the first negative sample document introduced during contrastive learning can be used to capture the semantic representation of documents with the same identifier prefix, and the second negative sample document introduced can be used to capture the semantic representation of documents with different identifier prefixes. Based on the above two types of negative sample documents, two corresponding contrast losses are applied to maximize the semantic score between the query and the positive document to achieve their similar representation.

[0120] The generative retrieval model (i.e., the second generative retrieval model) obtained through the above two-stage training can be configured in an electronic device for information retrieval. Exemplarily, the second generative retrieval model can be applied to a search advertising system, and the second generative retrieval model can retrieve and generate multiple relevant mini-program advertisements based on the query terms entered by the user in the search box of the application page.

[0121] The following is an exemplary description of the information retrieval method based on the second generative retrieval model.

[0122] like Figure 5 As shown, the information retrieval method provided by one or more embodiments of this specification may include the following steps:

[0123] Step S202, receiving an input query term.

[0124] Step S204: using the second generative retrieval model, generate the first k document identifiers corresponding to the query terms and the probability values ​​corresponding to the first k document identifiers respectively; wherein k is a positive integer greater than or equal to 1.

[0125] Taking the application of the second generative retrieval model to the search advertising system as an example, a user can enter a query term through the search box of the application page of the electronic device. After the electronic device receives the input query term, it can input the query term into the second generative retrieval model, and output the first k document identifiers and the probability values ​​corresponding to the first k document identifiers through the second generative retrieval model. For example, the second generative retrieval model can be used to generate a ranking list of the first k document identifiers and the probability values ​​corresponding to each document identifier using a beam search method.

[0126] For example, Figure 6As shown, after the query term "who is the best olympic gymnast of all time?" is input into the second generative retrieval model, the second generative retrieval model outputs the first three document identifiers olympic-gymnasts, gymnasts-olympic and gymnast-olympian. Among them, the probability value of the document identifier olympic-gymnasts is 0.85, the probability value of the document identifier gymnasts-olympic is 0.81, and the probability value of the document identifier gymnast-olympian is 0.78.

[0127] Step S206: determine the candidate documents corresponding to the first k document identifiers.

[0128] Among them, a document identifier may correspond to one or more candidate documents.

[0129] For example, Figure 6 As shown, the candidate document corresponding to the document identifier olympic-gymnasts is doc_108101; the document identifier gymnasts-olympic corresponds to three candidate documents, namely doc_67652, doc_109303 and doc_21876; the candidate document corresponding to the document identifier gymnast-olympian is doc_103098.

[0130] Step S208 : determining a semantic score corresponding to each candidate document based on the query semantic representation of the query term and the document semantic representation of each candidate document.

[0131] In one or more embodiments of the present specification, the query semantic representation and the document semantic representation may be generated by a second generative retrieval model.

[0132] In a possible implementation, after determining the candidate documents corresponding to the first k document identifiers, the query terms and the candidate documents are input into a second generative retrieval model, and the query semantic representation and the document semantic representation are output through the second generative retrieval model.

[0133] In a possible implementation, after the second generative retrieval model is trained, the second generative retrieval model can be used to generate document semantic representations corresponding to each document in the document library in advance. Each document semantic representation is associated with its corresponding document and saved in the database. In this way, after determining the candidate documents corresponding to the first k document identifiers, the document semantic representations corresponding to each candidate document can be directly obtained from the database.

[0134] Afterwards, the semantic score corresponding to each candidate document can be calculated using formula (2).

[0135] For example, the semantic score corresponding to the candidate document doc_108101 is 0.75, the semantic score corresponding to the candidate document doc_67652 is 0.70, the semantic score corresponding to the candidate document doc_109303 is 0.82, the semantic score corresponding to the candidate document doc_21876 is 0.73, and the semantic score corresponding to the candidate document doc_103098 is 0.68.

[0136] Step S210: determining a relevance score corresponding to each candidate document based on the probability value and semantic score corresponding to each candidate document.

[0137] For example, the following formula (5) can be used to calculate the relevance score of the candidate document:

[0138] rel(q, d) = P(z d |q)·s(q, d) Formula (5)

[0139] Among them, rel(q, d) represents the relevance score of candidate document d corresponding to query term q, P(z d |q) represents the probability value of the document identifier corresponding to the candidate document d, and s(q,d) represents the semantic score between the query term q and the candidate document d.

[0140] For example, Figure 6 As shown, according to the above formula (5), it can be calculated that the relevance score corresponding to the candidate document doc_108101 is 0.6642, the relevance score corresponding to the candidate document doc_67652 is 0.6375, the relevance score corresponding to the candidate document doc_109303 is 0.5913, the relevance score corresponding to the candidate document doc_21876 is 0.5610, and the relevance score corresponding to the candidate document doc_103098 is 0.5304.

[0141] Step S212: sort the candidate documents based on the relevance scores, and use the sorted candidate documents as retrieval results.

[0142] Since the relevance score incorporates the generation probability of the document identifier and the semantic score, the retrieval results obtained by sorting based on the relevance score take into account both the association between the query terms and the document identifier and the association between the query terms and the document, making the retrieval results more accurate. In addition, when the same document identifier corresponds to multiple candidate documents, the semantic score can be used to solve the problem of inaccurate candidate document sorting due to document identifier conflicts.

[0143] For example, Figure 6 As shown, if the candidate documents are ranked only by the generation probability of the document identifier, the ranking among the candidate documents doc_67652, doc_109303 and doc_21876 cannot be determined. However, based on the relevance score, the ranking of each candidate document can be determined.

[0144] Thus, the information retrieval method provided by one or more embodiments of the present specification can collaboratively determine the relevance between the query and the document by integrating the generation probability and semantic score of the document identifier. Thus, the documents corresponding to the generated identifiers are sorted based on the relevance score, which can improve the information retrieval effect.

[0145] It should be noted that, taking the second generative retrieval model applied to the search advertising system as an example, the search results obtained in step S212 may be ranked advertisements. The ranked advertisements are the advertisement recall results, and the search advertising system may perform subsequent rough sorting, fine sorting, and other processing based on the advertisement recall results.

[0146] It should also be noted that one or more embodiments of this specification only use the above-mentioned information retrieval method applied to a search advertising system as an example for illustrative description, and does not mean that there is no limitation on the specific application scenario. The information retrieval method provided in one or more embodiments of this specification can be applied to any scenario where documents related to a query need to be retrieved based on a query.

[0147] It can be understood that the above embodiments are only examples and can be modified in actual implementation. Those skilled in the art can understand that the modification methods of the above embodiments without creative labor fall within the protection scope of one or more embodiments of this specification and will not be repeated in the embodiments.

[0148] Based on the same inventive concept, one or more embodiments of the present specification also provide a training device for a generative retrieval model. Since the principle of the problem solved by the training device for the generative retrieval model is similar to that of the aforementioned training method for the generative retrieval model, the implementation of the training device for the generative retrieval model can refer to the implementation of the aforementioned training method for the generative retrieval model, and the repeated parts will not be repeated.

[0149] See also Figure 7 , Figure 7 A structural block diagram of a training device for a generative retrieval model provided in one or more embodiments of this specification. Figure 7 As shown, the training device 300 of the generative retrieval model may include: a first acquisition module 301, a first generation module 302, a first training module 303, a second acquisition module 304 and a second training module 305.

[0150] The first acquisition module 301 may be used to acquire first training data, where the first training data includes a one-to-one correspondence between a first query term and a first positive sample document.

[0151] The first generating module 302 may be used to generate first standard document identifiers corresponding to the first positive sample documents respectively;

[0152] The first training module 303 can be used to train the generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers using the one-to-one correspondence between the first query terms and the first standard document identifiers, thereby obtaining a first generative retrieval model;

[0153] A second acquisition module 304 may be used to acquire second training data, where the second training data includes a one-to-one correspondence between a second query term, a second positive sample document, and a negative sample document;

[0154] The second training module 305 can be used to train the first generative retrieval model to learn the correlation between the semantic representation of the query term and the semantic representation of the document by using the second training data through a comparative learning method to obtain a second generative retrieval model.

[0155] In one possible implementation, the first training module 303 is specifically used to: sample the corresponding first positive document fragments from each first positive sample document respectively; input the first query term and the first positive document fragment into the generative retrieval model to be trained respectively; output the first predicted document identifier corresponding to the first query term and the second predicted document identifier corresponding to the first positive document fragment through the generative retrieval model to be trained; determine the first generation loss based on the first standard document identifier, the first predicted document identifier and the second predicted document identifier; train the generative retrieval model to be trained until convergence based on the first generation loss to obtain a first generative retrieval model.

[0156] In one possible implementation, the negative sample document includes a first negative sample document and a second negative sample document; wherein the identifier prefix of the first negative sample document is partially or completely the same as the identifier prefix of the second positive sample document; and the identifier prefix of the second negative sample document is different from the identifier prefix of the second positive sample document.

[0157] In one possible implementation, the device also includes: a first negative sample document sampling module, used to generate a second standard document identifier corresponding to the second positive sample document; and sample a first preset number of documents from the document library as first negative sample documents in order from most to least number of documents having the same identifier prefix as the second standard document identifier.

[0158] In one possible implementation, the device also includes: a second negative sample document sampling module, which is used to input the second query term into the first generative retrieval model; output multiple alternative document identifiers related to the second query term through the first generative retrieval model; generate a second standard document identifier corresponding to the second positive sample document; and sample a second preset number of documents from the documents corresponding to the alternative document identifiers that are different from the second standard document identifier as second negative sample documents.

[0159] In one possible implementation, the first generative retrieval model includes an encoder; the second training module 305 is specifically used to: input the second query term, the second positive sample document, the first negative sample document, and the second negative sample document into the first generative retrieval model respectively; output the query semantic representation corresponding to the second query term, the positive sample semantic representation corresponding to the second positive sample document, the first negative sample semantic representation corresponding to the first negative sample document, and the second negative sample semantic representation corresponding to the second negative sample document through the encoder; determine the positive sample semantic score corresponding to the positive sample semantic representation, the first negative sample semantic score corresponding to the first negative sample semantic representation, and the second negative sample semantic score corresponding to the second negative sample semantic representation based on the query semantic representation; determine the contrast loss based on the positive sample semantic score, the first negative sample semantic score, and the second negative sample semantic score; train the first generative retrieval model to convergence based on the contrast loss to obtain the second generative retrieval model.

[0160] In one possible implementation, the first generative retrieval model also includes a decoder, a second training module 305, which is specifically used to: sample corresponding second positive document fragments from each second positive sample document; generate third standard document identifiers corresponding to the second positive sample document, the first negative sample document, and the second negative sample document; input the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document into the first generative retrieval model respectively; output the third predicted document identifiers corresponding to the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document respectively through the decoder; determine the second generation loss based on the third standard document identifiers and the third predicted document identifiers corresponding to the second query term, the second positive document fragment, the first negative sample document, and the second negative sample document respectively; based on the second generation loss and the contrast loss, train the first generative retrieval model to convergence to obtain the second generative retrieval model.

[0161] One or more embodiments of the present specification also provide an information retrieval device. Since the principle of solving the problem by the information retrieval device is similar to that of the aforementioned information retrieval method, the implementation of the information retrieval device can refer to the implementation of the aforementioned information retrieval method, and the repeated parts will not be repeated.

[0162] See also Figure 8 , Figure 8This is a structural block diagram of an information retrieval device provided in one or more embodiments of this specification. Figure 8 As shown, the information retrieval device 400 may include: a receiving module 401, a second generating module 402, a first determining module 403, a second determining module 404, a third determining module 405 and a sorting module 406.

[0163] The receiving module 401 may be used to receive an input query term.

[0164] The second generation module 402 can be used to generate the first k document identifiers corresponding to the query terms and the probability values ​​corresponding to the first k document identifiers respectively using the trained second generative retrieval model; wherein k is a positive integer greater than or equal to 1.

[0165] The first determination module 403 may be used to determine candidate documents corresponding to the first k document identifiers.

[0166] The second determination module 404 can be used to determine the semantic score corresponding to each candidate document based on the query semantic representation of the query term and the document semantic representation of each candidate document; wherein the query semantic representation and the document semantic representation are obtained through the second generative retrieval model.

[0167] The third determination module 405 may be used to determine the relevance score corresponding to each candidate document based on the probability value and the semantic score corresponding to each candidate document.

[0168] The sorting module 406 may be used to sort the candidate documents based on the relevance scores and use the sorted candidate documents as the retrieval results.

[0169] In a possible implementation, the query term is input information received through a search box on an application page of an electronic device, and the candidate document is an advertisement.

[0170] See also Fig. 9 , Fig. 9 This is a structural block diagram of an electronic device provided in one or more embodiments of this specification. Fig. 9 As shown, the electronic device 500 may include a processor 501 and a memory 502; the memory 502 may be coupled to the processor 501. It is worth noting that the Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0171] In a possible implementation, the functions of the generative retrieval model training device 300 or the information retrieval device 400 may be integrated into the processor 501. The processor 501 may be configured to perform the following operations:

[0172] Acquire first training data, the first training data including one-to-one corresponding first query terms and first positive sample documents; generate first standard document identifiers corresponding to each first positive sample document; use the one-to-one corresponding first query terms and first standard document identifiers to train a generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers, and obtain a first generative retrieval model; acquire second training data, the second training data including one-to-one corresponding second query terms, second positive sample documents and negative sample documents; use the second training data to train the first generative retrieval model through a comparative learning method to learn the correlation between the semantic representation of the query terms and the semantic representation of the document, and obtain a second generative retrieval model.

[0173] Alternatively, the processor 501 may be configured to perform the following operations:

[0174] Receive an input query term; generate the first k document identifiers corresponding to the query term and the probability values ​​corresponding to the first k document identifiers by using the trained second generative retrieval model; wherein k is a positive integer greater than or equal to 1; determine the candidate documents corresponding to the first k document identifiers; determine the semantic score corresponding to each candidate document based on the query semantic representation of the query term and the document semantic representation of each candidate document; wherein the query semantic representation and the document semantic representation are obtained by the second generative retrieval model; determine the relevance score corresponding to each candidate document based on the probability value and the semantic score corresponding to each candidate document; sort the candidate documents based on the relevance score, and use the sorted candidate documents as the retrieval result.

[0175] In another possible implementation, the generative retrieval model training device 300 or the information retrieval device 400 may be configured separately from the processor 501. For example, the generative retrieval model training device 300 or the information retrieval device 400 may be configured as a chip connected to the processor 501, and the generative retrieval model training or information retrieval may be implemented through the control of the processor 501.

[0176] In addition, in some optional implementations, the electronic device 500 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 500 does not necessarily have to include Figure 5 In addition, the electronic device 500 may also include Figure 5 For components not shown, reference may be made to the prior art.

[0177] In some optional implementations, the processor 501 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 500.

[0178] The memory 502 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The memory 502 may store the information related to the training device 300 or the information retrieval device 400 of the generative retrieval model, and may also store a program for executing the related information. The processor 501 may execute the program stored in the memory 502 to implement information storage or processing.

[0179] The input unit can provide input to the processor 501. The input unit is, for example, a key or a touch input device. The power supply can be used to provide power to the electronic device 500. The display can be used to display display objects such as images and text. The display can be, for example, an LCD display, but is not limited thereto.

[0180] The memory 502 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 502 may also be some other type of device. The memory 502 includes a buffer memory (sometimes referred to as a buffer). The memory 502 may include an application / function storage unit for storing application programs and function programs or processes for executing the operation of the electronic device 500 through the processor 501.

[0181] The memory 502 may also include a data storage unit for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit of the memory 502 may include various drivers for the communication function of the computer device and / or for executing other functions of the computer device (such as a messaging application, a contact book application, etc.).

[0182] The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor 501 to provide input signals and receive output signals, which may be the same as in a conventional mobile communication terminal.

[0183] Based on different communication technologies, multiple communication modules may be provided in the same computer device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to the processor 501, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.

[0184] One or more embodiments of the present specification also provide a computer-readable storage medium capable of implementing all the steps in the training method of the generative retrieval model or the information retrieval method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the training method of the generative retrieval model or the information retrieval method in the above embodiments is implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0185] Acquire first training data, the first training data including one-to-one corresponding first query terms and first positive sample documents; generate first standard document identifiers corresponding to each first positive sample document; use the one-to-one corresponding first query terms and first standard document identifiers to train a generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers, and obtain a first generative retrieval model; acquire second training data, the second training data including one-to-one corresponding second query terms, second positive sample documents and negative sample documents; use the second training data to train the first generative retrieval model through a comparative learning method to learn the correlation between the semantic representation of the query terms and the semantic representation of the document, and obtain a second generative retrieval model.

[0186] Alternatively, the processor implements the following steps when executing the computer program:

[0187] Receive an input query term; generate the first k document identifiers corresponding to the query term and the probability values ​​corresponding to the first k document identifiers by using the trained second generative retrieval model; wherein k is a positive integer greater than or equal to 1; determine the candidate documents corresponding to the first k document identifiers; determine the semantic score corresponding to each candidate document based on the query semantic representation of the query term and the document semantic representation of each candidate document; wherein the query semantic representation and the document semantic representation are obtained by the second generative retrieval model; determine the relevance score corresponding to each candidate document based on the probability value and the semantic score corresponding to each candidate document; sort the candidate documents based on the relevance score, and use the sorted candidate documents as the retrieval result.

[0188] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0189] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, devices (systems) or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0190] One or more embodiments of the present specification are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0194] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meaning of the above terms in one or more embodiments of this specification can be understood according to the specific circumstances.

[0195] It should be noted that, in the absence of conflict, one or more embodiments of this specification and the features in the embodiments may be combined with each other. One or more embodiments of this specification are not limited to any single aspect, nor to any single embodiment, nor to any combination and / or replacement of these aspects and / or embodiments. Moreover, each aspect and / or embodiment of one or more embodiments of this specification may be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of this specification, rather than to limit them. Although one or more embodiments of this specification have been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of one or more embodiments of this specification, and they should all be included in the scope of the claims and the specification of one or more embodiments of this specification.

[0197] One or more embodiments of this specification are described above in conjunction with optional implementation methods, but these implementation methods are only exemplary and serve only as an illustration. On this basis, multiple replacements and improvements can be made to one or more embodiments of this specification, all of which fall within the scope of protection of one or more embodiments of this specification.

Claims

1. A training method for a generative retrieval model, characterized in that: The method comprises: Acquire first training data, where the first training data includes a first query term and a first positive sample document in one-to-one correspondence; Generate first standard document identifiers corresponding to each of the first positive sample documents; Using the one-to-one correspondence between the first query terms and the first standard document identifier, training the generative retrieval model to be trained to learn the correlation between the query terms and the document identifier, thereby obtaining a first generative retrieval model; Acquire second training data, where the second training data includes a second query term, a second positive sample document, and a negative sample document in one-to-one correspondence; By using a comparative learning method, the first generative retrieval model is trained with the second training data to learn the correlation between the semantic representation of the query terms and the semantic representation of the document, thereby obtaining a second generative retrieval model.

2. The method according to claim 1, characterized in that The method of training the generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers by using the one-to-one correspondence between the first query terms and the first standard document identifiers to obtain the first generative retrieval model includes: Sampling corresponding first positive document segments from each of the first positive sample documents respectively; respectively inputting the first query term and the first positive document fragment into a generative retrieval model to be trained; Outputting a first predicted document identifier corresponding to the first query term and a second predicted document identifier corresponding to the first positive document fragment through the generative retrieval model to be trained; determining a first generation loss based on the first standard document identifier, the first prediction document identifier, and the second prediction document identifier; The generative retrieval model to be trained is trained to convergence based on the first generation loss to obtain a first generative retrieval model.

3. The method according to claim 1, characterized in that The negative sample documents include a first negative sample document and a second negative sample document; wherein the identifier prefix of the first negative sample document is partially or completely the same as the identifier prefix of the second positive sample document; and the identifier prefix of the second negative sample document is different from the identifier prefix of the second positive sample document.

4. The method according to claim 3, characterized in that The method further comprises: Generating a second standard document identifier corresponding to the second positive sample document; A first preset number of documents are sampled from the document library as first negative sample documents in descending order of the number of documents having the same identifier prefix as the second standard document identifier.

5. The method according to claim 3, characterized in that The method further comprises: inputting the second query term into the first generative retrieval model; Outputting a plurality of candidate document identifiers related to the second query term through the first generative retrieval model; Generating a second standard document identifier corresponding to the second positive sample document; A second preset number of documents are sampled from documents corresponding to candidate document identifiers different from the second standard document identifier as second negative sample documents.

6. The method according to claim 3, characterized in that The first generative retrieval model includes an encoder; the second training data is used to train the first generative retrieval model to learn the correlation between the semantic representation of the query term and the semantic representation of the document by using a contrastive learning method to obtain a second generative retrieval model, including: Inputting the second query term, the second positive sample document, the first negative sample document and the second negative sample document into the first generative retrieval model respectively; Outputting, by the encoder, a query semantic representation corresponding to the second query term, a positive sample semantic representation corresponding to the second positive sample document, a first negative sample semantic representation corresponding to the first negative sample document, and a second negative sample semantic representation corresponding to the second negative sample document; Determine, based on the query semantic representation, a positive sample semantic score corresponding to the positive sample semantic representation, a first negative sample semantic score corresponding to the first negative sample semantic representation, and a second negative sample semantic score corresponding to the second negative sample semantic representation; Determining a contrast loss based on the positive sample semantic score, the first negative sample semantic score, and the second negative sample semantic score; The first generative retrieval model is trained to convergence based on the contrast loss to obtain a second generative retrieval model.

7. The method according to claim 6, characterized in that The first generative retrieval model further includes a decoder, and the first generative retrieval model is trained to convergence based on the contrast loss to obtain a second generative retrieval model, including: Sampling corresponding second positive document segments from each of the second positive sample documents respectively; Generating third standard document identifiers corresponding to the second positive sample document, the first negative sample document, and the second negative sample document respectively; inputting the second query term, the second positive document segment, the first negative sample document and the second negative sample document into the first generative retrieval model respectively; Outputting, by the decoder, third predicted document identifiers corresponding to the second query term, the second positive document segment, the first negative sample document, and the second negative sample document respectively; Determining a second generation loss based on a third standard document identifier and a third predicted document identifier corresponding to the second query term, the second positive document segment, the first negative sample document, and the second negative sample document, respectively; Based on the second generative loss and the contrastive loss, the first generative retrieval model is trained until convergence to obtain a second generative retrieval model.

8. An information retrieval method, characterized in that: Applied to electronic equipment, the method comprises: Receive input query terms; Using the second generative retrieval model trained by the method according to any one of claims 1 to 7, generating the first k document identifiers corresponding to the query terms and the probability values ​​corresponding to the first k document identifiers respectively; wherein k is a positive integer greater than or equal to 1; Determine candidate documents corresponding to the first k document identifiers respectively; Determine the semantic score corresponding to each of the candidate documents based on the query semantic representation of the query term and the document semantic representation of each of the candidate documents; wherein the query semantic representation and the document semantic representation are obtained by the second generative retrieval model; Determine a relevance score corresponding to each of the candidate documents based on the probability value and the semantic score corresponding to each of the candidate documents; The candidate documents are sorted based on the relevance scores, and the sorted candidate documents are used as retrieval results.

9. The method according to claim 8, characterized in that The query term is input information received through a search box on an application page of the electronic device, and the candidate document is an advertisement.

10. A training device for a generative retrieval model, characterized in that: The device comprises: A first acquisition module, configured to acquire first training data, wherein the first training data includes a first query term and a first positive sample document in one-to-one correspondence; A first generating module, used to generate first standard document identifiers corresponding to each of the first positive sample documents; A first training module is used to train the generative retrieval model to be trained to learn the correlation between the query terms and the document identifiers using the one-to-one correspondence between the first query terms and the first standard document identifiers, so as to obtain a first generative retrieval model; A second acquisition module, used to acquire second training data, wherein the second training data includes a second query term, a second positive sample document, and a negative sample document in one-to-one correspondence; The second training module is used to train the first generative retrieval model by using the second training data through a comparative learning method to learn the correlation between the semantic representation of the query term and the semantic representation of the document, so as to obtain a second generative retrieval model.

11. The device according to claim 10, characterized in that The first training module is specifically used for: Sampling corresponding first positive document segments from each of the first positive sample documents respectively; respectively inputting the first query term and the first positive document fragment into a generative retrieval model to be trained; Outputting a first predicted document identifier corresponding to the first query term and a second predicted document identifier corresponding to the first positive document fragment through the generative retrieval model to be trained; determining a first generation loss based on the first standard document identifier, the first prediction document identifier, and the second prediction document identifier; The generative retrieval model to be trained is trained to convergence based on the first generation loss to obtain a first generative retrieval model.

12. The device according to claim 10, characterized in that The negative sample documents include a first negative sample document and a second negative sample document; wherein the identifier prefix of the first negative sample document is partially or completely the same as the identifier prefix of the second positive sample document; and the identifier prefix of the second negative sample document is different from the identifier prefix of the second positive sample document.

13. The device according to claim 12, characterized in that The device also includes: The first negative sample document sampling module is used to generate a second standard document identifier corresponding to the second positive sample document; and sample a first preset number of documents from the document library as the first negative sample documents in descending order of the number of documents having the same identifier prefix as the second standard document identifier.

14. The device according to claim 12, characterized in that The device also includes: A second negative sample document sampling module is used to input the second query term into the first generative retrieval model; output multiple alternative document identifiers related to the second query term through the first generative retrieval model; generate a second standard document identifier corresponding to the second positive sample document; and sample a second preset number of documents from the documents corresponding to the alternative document identifiers that are different from the second standard document identifier as second negative sample documents.

15. The device according to claim 12, characterized in that The first generative retrieval model includes an encoder; the second training module is specifically used for: Inputting the second query term, the second positive sample document, the first negative sample document and the second negative sample document into the first generative retrieval model respectively; Outputting, by the encoder, a query semantic representation corresponding to the second query term, a positive sample semantic representation corresponding to the second positive sample document, a first negative sample semantic representation corresponding to the first negative sample document, and a second negative sample semantic representation corresponding to the second negative sample document; Determine, based on the query semantic representation, a positive sample semantic score corresponding to the positive sample semantic representation, a first negative sample semantic score corresponding to the first negative sample semantic representation, and a second negative sample semantic score corresponding to the second negative sample semantic representation; Determining a contrast loss based on the positive sample semantic score, the first negative sample semantic score, and the second negative sample semantic score; The first generative retrieval model is trained to convergence based on the contrast loss to obtain a second generative retrieval model.

16. The device according to claim 15, characterized in that The first generative retrieval model further includes a decoder, and the second training module is specifically used to: Sampling corresponding second positive document segments from each of the second positive sample documents respectively; Generating third standard document identifiers corresponding to the second positive sample document, the first negative sample document, and the second negative sample document respectively; inputting the second query term, the second positive document segment, the first negative sample document and the second negative sample document into the first generative retrieval model respectively; Outputting, by the decoder, third predicted document identifiers corresponding to the second query term, the second positive document segment, the first negative sample document, and the second negative sample document respectively; Determining a second generation loss based on a third standard document identifier and a third predicted document identifier corresponding to the second query term, the second positive document segment, the first negative sample document, and the second negative sample document, respectively; Based on the second generative loss and the contrastive loss, the first generative retrieval model is trained until convergence to obtain a second generative retrieval model.

17. An information retrieval device, characterized in that: Applied to electronic equipment, the device comprises: A receiving module, used for receiving an input query term; A second generation module, configured to generate the first k document identifiers corresponding to the query terms and the probability values ​​corresponding to the first k document identifiers respectively using the second generative retrieval model trained by the method according to any one of claims 1 to 7; wherein k is a positive integer greater than or equal to 1; A first determination module is used to determine candidate documents corresponding to the first k document identifiers respectively; A second determination module is used to determine the semantic scores corresponding to the candidate documents based on the query semantic representation of the query term and the document semantic representation of each candidate document; wherein the query semantic representation and the document semantic representation are obtained by the second generative retrieval model; A third determination module, configured to determine a relevance score corresponding to each of the candidate documents based on the probability value and the semantic score corresponding to each of the candidate documents; A sorting module is used to sort the candidate documents based on the relevance scores and use the sorted candidate documents as retrieval results.

18. The device according to claim 17, characterized in that The query term is input information received through a search box on an application page of the electronic device, and the candidate document is an advertisement.

19. An electronic device, characterized in that: The electronic device comprises: a memory for storing a computer program product; A processor, configured to execute a computer program product stored in the memory, and when the computer program product is executed, implements the method described in any one of claims 1 to 7 or 8 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed, implement the method described in any one of claims 1 to 7 or 8 to 9.

Images