Conversational Information Retrieval Methods Based on Pre-trained Language Models

Abstract

This invention relates to the field of information retrieval methods, and discloses a conversational information retrieval method based on a pre-trained language model. By filtering historical query information related to prepositions and employing a dual-tower fine-grained semantic interaction model, it solves the problem that existing retrieval techniques tend to ignore semantic relationships, resulting in insufficient accuracy of query results.

Description

Technical Field

[0001] This invention relates to the field of information retrieval methods, and in particular to a conversational information retrieval method based on a pre-trained language model. Background Technology

[0002] The widespread adoption of next-generation conversational assistants (such as Alexa, Siri, Cortana, Bixby, and Google Assistant) has broadened the application scenarios of conversational information retrieval methods and increased the importance of conversational information retrieval technology. The goal of conversational information retrieval is for the conversational model to understand user behavior during the interactive search process and express the shifts in user needs based on query rounds. Simultaneously, information needs are characterized by complexity (requiring multiple rounds of refinement), diversity (spanning different information categories), open domain (not requiring access to expert domain knowledge), and answerability (sufficient coverage within the text collection). Conversational information retrieval is one of the directions for next-generation information retrieval proposed at the 2020 International Conference on Text Information Retrieval. It clarifies the characteristics of conversational information retrieval: it is favored for its ability to meet users' complex information needs, and it provides users with convenient and accurate information access through conversational interfaces and portable devices.

[0003] Currently, there are two main solutions for conversational retrieval. The first uses a generative method with GPT-3 to rewrite the conversational query into a separate, context-independent query. The second method represents both the query and the document as dense vector representations. The representation of the conversational query is obtained by concatenating all historical queries and the current query. The document representation is derived by using the document representation from previous ad hoc retrievals, as there is little change between them. Finally, dense vector retrieval is used. However, not all historical queries are useful for the current query. Therefore, we formulate rules to select useful historical information and build a model using contrastive learning methods. This allows our combined query representation to continuously approximate the manually rewritten query, resulting in a less noisy and more useful query encoding representation, which helps the conversational retrieval model find documents relevant to the context of the current query.

[0004] Chinese patent CN202110795247.7 discloses a "Pre-trained Dual Attention Neural Network Semantic Inference Dialogue Retrieval Method and System, Retrieval Device, and Storage Medium," which uses a BM25 model based on the binary independence assumption, assuming that terms are mutually independent. It measures the relevance between the query and the document by calculating the frequency of the query term in the document and the "rarity" (inverse document frequency) of the query term among numerous documents. However, the binary independence assumption is inaccurate because the semantics of terms are context-dependent, not independent. This scheme fails to filter out information useful for the current query through contextual information, resulting in irrelevant information in the retrieval results and reduced accuracy. Because some irrelevant information supports the binary independence assumption, it is difficult to further optimize it. Summary of the Invention

[0005] Existing conversational information retrieval technologies concatenate all historical queries as input to consider contextual information. However, not all historical information is useful for the current query; irrelevant information introduces noise into the model, leading to a decrease in the accuracy of retrieval results. To address these issues, this invention proposes a conversational information retrieval method based on a pre-trained language model to improve the accuracy of information retrieval results.

[0006] The technical solution of the present invention includes the following steps:

[0007] S1: Obtain the encoded representation of the document using the existing text representation model BERT.

[0008] S2: For a set of dialog queries Regarding the first Round query Find historical queries related to this query requirement. The two are concatenated and then input into the text representation model BERT.

[0009] S3: Through comparative learning, the encoded representation of queries constructed by the learned model is made close to the encoded representation of manually rewritten queries.

[0010] S4: Each query statement in a set of conversational queries and its related historical queries are concatenated and input into the model trained in S3 for encoding. The semantic similarity with the document encoding representation obtained in S1 is calculated, and the documents are sorted from largest to smallest.

[0011] S5: Construct a dual-tower fine-grained semantic interaction model using a contrastive learning method, train it using the set constructed in S4, and calculate the ranking loss of the model using cross-entropy loss.

[0012] S6: Use the model trained in S5 to retrieve queries on the test set and obtain the ranking results.

[0013] As a preferred option, the historical queries related to this query requirement described in S2 are... Historical information is selected according to the following rules: For the current query If the pronouns "it", "he", "she", "they", or "them" appear, then the current query is compared with the previous query history. splicing, and additional checks and queries. If the words "it", "he", "she", "they", or "them" also appear, then continue tracing back to the previous historical information and include the previous historical query. Combine it with the current query. If it doesn't exist, don't trace back any further.

[0014] in For query The query can be performed on the relevant historical rounds. For manually rewritten queries, a single query record is sufficient to obtain the encoded representation, as shown below:

[0015] This approach selects only sentences containing relevant keywords, rather than all sentences, and filters from historical information, thus improving accuracy. Furthermore, since this approach only uses relevant information, the number of samples required to achieve the target accuracy is less than that of the approach that selects all sentences, making it more in line with the needs of conversational information retrieval.

[0016] Furthermore, in formula S2

[0017]

[0018]

[0019] The encoded representation of queries and documents is the output (hidden_states) of each node in the hidden layer. The output of the hidden layer is preserved.

[0020] This step is used to capture fine-grained semantic similarity during semantic interaction. It also addresses the issue that the CLS results (pooler_output) are not suitable for text representation.

[0021] As a preferred embodiment, the loss function described in S3 is specifically:

[0022]

[0023] Where batch_size is the maximum allowed input length for the query. For manually rewritten queries The encoding representation, This is the encoded representation of the query.

[0024] This loss function represents the difference between the query representation and the manually rewritten query encoding representation. By reducing this difference, the training model is refined. Contrastive learning is used to refine the encoding representation of the manually rewritten query. and the encoded representation of the constructed query The models are trained separately so that the encoded representation of the queries generated by the trained models approximates the encoded representation of the manually rewritten queries.

[0025] Preferably, S4 uses the model trained in S3 to perform conversational retrieval, sorts the query results, extracts the N most similar documents from the query results, labels these documents, and selects positive examples relevant to the query and negative examples irrelevant to the query from the top n results with the closest ranking to construct the training dataset for the ranking model. Based on this result, the triples for the ranking model are constructed:

[0026]

[0027] in To mark as and query Related documents, and A set of negatively correlated documents.

[0028] Positive samples and negative samples are similar but different. This method reduces the impact of noise on the model and improves the model's discrimination ability.

[0029] Furthermore, the number of negative examples that are not relevant to the query should be controlled to within 200, preferably 50-100.

[0030] While a training dataset for a ranking model can be constructed as long as the selected n results contain both positive and negative examples, and are not all entirely relevant to the query or all entirely irrelevant, theoretically, more data leads to better results. However, results with lower similarity are inevitably less relevant, causing data bias. Because bias affects the reliability of the training dataset, the number of negative examples should be below 200; selecting 50-100 negative examples yields better results.

[0031] As a preferred option, S5 utilizes BERT to construct a dual-tower semantic matching model, and uses the training set constructed in S4 to train model M3. The model ranking loss is calculated using cross-entropy loss as follows:

[0032]

[0033] in and These are queries The encoding representation of related and unrelated documents. Indicates the similarity between the query and the positive example document. This indicates the similarity between the query and the negative example document.

[0034] As a preferred option, S6 uses a trained model to perform conversational retrieval. Query and documents The semantic similarity is calculated as follows: First, the sum of the semantic similarities of the words in the document most similar to each query term is calculated, which helps capture fine-grained relevance to the query. Second, the similarity of the average vector representations of the query and the document is calculated. The two are then added together to obtain the semantic similarity between the query and the document. The calculation method is as follows:

[0035]

[0036] in For input query The The encoded representation of each query term, For document The The encoding representation of each word. To query the number of tokens. This represents the number of document tokens.

[0037] The results are sorted by similarity scores to obtain optimized conversational search results.

[0038] In real-world scenarios, information that is "most similar" to the query semantics is the most important. This technical solution breaks away from the technical bias of existing technologies that rely on assumptions such as binary independence to achieve theoretically higher accuracy. By starting with information that is semantically most similar, it improves the accuracy of the results. Detailed Implementation

[0039] Example 1

[0040] The specific implementation of the conversational information retrieval method based on pre-trained language models includes the following steps:

[0041] S1: Obtain the encoded representation of the document using the existing text representation model BERT.

[0042] S2: For a set of dialog queries Regarding the first Round query Find historical queries related to this query requirement. The two are concatenated and then input into the text representation model BERT.

[0043] The historical queries related to this query request described in S2 Historical information is selected according to the following rules: For the current query If the pronouns "it", "he", "she", "they", or "them" appear, then the current query is compared with the previous query history. splicing, and additional checks and queries. If the words "it", "he", "she", "they", or "them" also appear, then continue tracing back to the previous historical information and include the previous historical query. Combine it with the current query. If it doesn't exist, don't trace back any further.

[0044]

[0045] in For query The query can be performed on the relevant historical rounds. For manually rewritten queries, a single query record is sufficient to obtain the encoded representation, as shown below:

[0046]

[0047] Among them, the formula in S2 is:

[0048]

[0049]

[0050] The encoded representation of queries and documents is the output (hidden_states) of each node in the hidden layer. The output of the hidden layer is preserved.

[0051] S3: Through comparative learning, the encoded representation of queries constructed by the learned model is made close to the encoded representation of manually rewritten queries.

[0052] The loss function used in this step is as follows:

[0053]

[0054] Where batch_size is the maximum allowed input length for the query. For manually rewritten queries The encoding representation, This is the encoded representation of the query.

[0055] This loss function represents the difference between the query representation and the manually rewritten query encoding representation. By reducing this difference, the training model is refined. Contrastive learning is used to refine the encoding representation of the manually rewritten query. and the encoded representation of the constructed query The models are trained separately so that the encoded representation of the queries generated by the trained models approximates the encoded representation of the manually rewritten queries.

[0056] S4: Each query statement in a set of conversational queries, along with its related historical queries, is concatenated and input into the model trained in S3 for encoding. The semantic similarity between this model and the document encoding obtained in S1 is calculated, and the documents are sorted from highest to lowest similarity. The N most similar documents are extracted from the query results, labeled, and from the top n closest results, positive examples relevant to the query and negative examples irrelevant to the query are selected to construct the training dataset for the ranking model. Based on this result, the triples for the ranking model are constructed:

[0057]

[0058] in To mark as and query Related documents, and A set of negatively correlated documents.

[0059] The number of negative instances that are irrelevant to the query should be kept below 200.

[0060] S5: Construct a dual-tower fine-grained semantic interaction model using a contrastive learning method, train it using the set constructed in S4, and calculate the ranking loss of the model using cross-entropy loss.

[0061] A dual-tower semantic matching model is constructed using BERT, and the model is trained using a training set built with S4. The model ranking loss is calculated using cross-entropy loss.

[0062]

[0063] in and These are queries The encoding representation of related and unrelated documents. Indicates the similarity between the query and the positive example document. This indicates the similarity between the query and the negative example document.

[0064] S6: Use the model trained in S5 to retrieve queries on the test set and obtain the ranking results.

[0065] Perform conversational retrieval using the trained model. Query and documents The semantic similarity is calculated as follows: First, the sum of the semantic similarities of the words in the document most similar to each query term is calculated, which helps capture fine-grained relevance to the query. Second, the similarity of the average vector representations of the query and the document is calculated. The two are then added together to obtain the semantic similarity between the query and the document. The calculation method is as follows:

[0066]

[0067] in For input query The The encoded representation of each query term, For document The The encoding representation of each word. To query the number of tokens. This represents the number of document tokens.

[0068] The results are sorted by similarity scores to obtain optimized conversational search results.

[0069] Example 2

[0070] According to the implementation method of the conversational information retrieval method based on a pre-trained language model described in Example 1, in step S4, the number of negative examples irrelevant to the query is 100. By selecting a sufficient number of negative examples irrelevant to the query, the bias caused by the naturally low relevance of irrelevant results can be reduced.

[0071] Example 3

[0072] According to the implementation method of the conversational information retrieval method based on a pre-trained language model described in Example 1, in step S4, the number of negative examples irrelevant to the query is 50. By reducing the number of negative examples irrelevant to the query, it is possible to better handle situations where the sample size is small and the number of irrelevant negative examples is insufficient.

Claims

1. A conversational information retrieval method based on a pre-trained language model, characterized in that: S1: Obtain the encoded representation of the document using the existing text representation model BERT; S2: For a set of dialog queries Regarding the first Round query Find historical queries related to this query requirement. ; Concatenate the two and input them into the text representation model BERT; For the current query If a pronoun appears, then the current query is compared with the previous historical query. splicing, and additional checks and queries. If similar pronouns appear, continue tracing back to the previous historical information and include the previous historical query. Combine it with the current query; if not found, do not trace back further. S3: Through comparative learning, the encoded representation of queries built by the learned model is close to the encoded representation of manually rewritten queries; S4: Each query statement in a set of conversational queries and its related historical queries are concatenated and input into the model trained in S3 for encoding. The semantic similarity with the document encoding representation obtained in S1 is calculated, and the documents are sorted from largest to smallest. S5: Construct a dual-tower fine-grained semantic interaction model using a contrastive learning method, train it using the set constructed in S4, and calculate the ranking loss of the model using cross-entropy loss. S6: Use the model trained in S5 to retrieve queries on the test set and obtain the ranking results.

2. The conversational information retrieval method based on a pre-trained language model according to claim 1, characterized in that: The historical queries related to this query request described in S2 Historical information is selected according to the following rules: in For query Batch queries of relevant historical cycles; For manually rewritten queries, a single query record is sufficient to obtain the encoded representation, as shown below: 。 3. The conversational information retrieval method based on a pre-trained language model according to claim 2, characterized in that: Formula in S2 The encoded representation of queries and documents is the output of each node in the hidden layer; the output of the hidden layer is preserved.

4. The conversational information retrieval method based on a pre-trained language model according to claim 1, characterized in that: The contrastive learning described in S3 has the following loss function: Where batch_size is the maximum allowed input length for the query. For manually rewritten queries The encoding representation, The encoded representation of the query; This loss function represents the difference between the query representation and the manually rewritten query encoding representation. By reducing this difference, the training model is refined. Contrastive learning is used to refine the encoding representation of the manually rewritten query. and the encoded representation of the constructed query The models are trained separately so that the encoded representation of the queries generated by the trained models approximates the encoded representation of the manually rewritten queries.

5. The conversational information retrieval method based on a pre-trained language model according to claim 1, characterized in that: S4 uses the model trained in S3 to perform conversational retrieval, sorts the query results, extracts the N most similar documents from the query results, labels the documents, and selects positive examples relevant to the query and negative examples irrelevant to the query from the top n results in the ranking to construct the training dataset for the ranking model; based on this result, it constructs the triples for the ranking model: in To mark as and query Related documents, and A set of negatively correlated documents.

6. The conversational information retrieval method based on a pre-trained language model according to claim 5, characterized in that: The number of negative instances that are irrelevant to the query should be controlled between 50 and 100.

7. The conversational information retrieval method based on a pre-trained language model according to claim 1, characterized in that: S5 uses BERT to build a dual-tower semantic matching model, and uses the training set built in S4 to train model M3; the model ranking loss is calculated using cross-entropy loss as follows: in and These are queries The encoding representation of related and unrelated documents; Indicates the similarity between the query and the positive example document. This indicates the similarity between the query and the negative example document.

8. The conversational information retrieval method based on a pre-trained language model according to claim 1, characterized in that: In S6, a pre-trained model is used to perform conversational retrieval; queries and documents The semantic similarity is calculated as follows: First, the sum of the semantic similarities of the words in the document most similar to each query term is calculated, which helps to capture fine-grained relevance to the query. Second, the similarity of the average vector representations of the query and the document is calculated. The two are then added together to obtain the semantic similarity between the query and the document. The calculation method is as follows: in For input query The The encoded representation of each query term, For document The The encoded representation of each word; To query the number of tokens; The number of document tokens; The results are sorted by similarity scores to obtain optimized conversational search results.

Images