A conversational retrieval method for modeling historical adjacent query changes

Abstract

This invention utilizes network search technology to implement a conversational retrieval method that models the evolution of historical adjacent queries. This method consists of four modules: a word weight calculation module, a representation-based matching module, a word-level interaction module, and a document score calculation module. The word weight calculation module models the evolving relationships between historical adjacent queries, calculates three word weights, and uses these three word weights to classify query variations. The results of the word weight calculation module are sent to the representation-based matching module for representation-based matching of candidate documents with the conversation context, and to the word-level interaction module for word-level interaction, respectively, to obtain matching and interaction scores. Finally, a fully connected layer in the document score calculation module calculates the score.

Description

Technical Field

[0001] The present invention relates to the field of network search technology, and in particular to a conversational retrieval method for modeling changes in historical adjacent queries. Background Art

[0002] There are currently some conversation-based retrieval models. Some models consider each query within a conversation separately and then combine the results of each query. In addition, more and more models consider that historical query sequences contain important information. When users ask new queries, they often reflect their query intent. Therefore, some models attempt to model query changes at the word level to improve conversational retrieval. With the widespread application of deep models, some models attempt to model query changes at the semantic level. Some of these models explicitly construct a distributed representation of query changes and then match this representation with candidate documents. Others use sequence modeling to describe the query change process, ultimately forming an implicit user intent representation, which is then interacted with candidate documents.

[0003] Existing models that improve conversational retrieval by modeling query changes only consider the change relationship between the current query and previous queries, while ignoring the change relationship between adjacent historical queries. This results in the omission of some user preference information contained in the evolution of historical queries. Summary of the Invention

[0004] To this end, the present invention first proposes a conversational retrieval method for modeling changes in historical adjacent queries. The method consists of four modules: a word weight calculation module, a representation-based matching module, a word-level interaction module, and a document score calculation module. The word weight calculation module models the change relationship of historical adjacent queries, calculates three word weights, and uses these three word weights to classify query changes. The result of the word weight calculation module is used in the representation-based matching module to perform representation-based matching between candidate documents and conversation contexts. At the same time, the word weights are also used in the word-level interaction module to perform fine-grained interaction at the word level, and obtain representation-based matching and fine-grained interaction scores respectively. Subsequently, a fully connected layer of the document score calculation module is used to calculate the document score for the search engine's document ranking. Finally, a multi-task learning strategy is used to simultaneously train the candidate document ranking task and the query change classification task.

[0005] The specific implementation of the word weight calculation module is as follows: for each word in a query or document, the semantic representation of each word including the context is calculated through Transformers:

[0006]

[0007]

[0008] where e q and e d are the initial word vectors of the words in the query and document, respectively, obtained through Fasttext training, e p is the positional encoding, Trm is the Transformers model,

[0009] For two adjacent historical queries q i-1 and q i , using the query change modeling unit to measure the change relationship between them through three word weights, namely q i-1 The removal weight of each word in q i The added weight of each word in and retention weights The query change modeling unit input includes not only the previous query and the next query, but also the user satisfaction click document corresponding to the previous query, that is, Based on these three different types of word weights, three corresponding query representations can be obtained, namely as well as These three query representations and the original word vector representations corresponding to the two queries are used as input, and the query change classifier is used to predict the change type corresponding to this pair of adjacent queries:

[0010]

[0011] After obtaining the three word weights, the representation-based matching module first uses a representation-based matching method to measure the degree of association between the candidate document and the conversation context. On the basis of the method of obtaining three query representations based on the three word weights, the query representation obtained by the original word vector representation is additionally introduced. For the above four query representations, Transformer is used to model the entire query. Specifically:

[0012] The representations of each query are interacted with each other through the transformer, and the representation corresponding to the last query is selected As the semantic representation of the entire query sequence, the other three types of representations can be obtained similarly

[0013]

[0014]

[0015]

[0016] By splicing these four representations together and passing them through a fully connected layer, we can obtain the semantic representation of the entire session. Compare this representation to the representation of the candidate document Calculate the similarity to get the matching score s based on the representation rep .

[0017] The word-level interaction module designs an attentive kernel pooling, which uses three word weights to extract three types of interaction features: Similarly, two other interaction features can be obtained through the other two word weights and

[0018]

[0019]

[0020] And use ordinary kernel pooling to obtain unweighted interaction features

[0021] These four types of matching features are passed through the fully connected layer to obtain four types of interaction scores:

[0022]

[0023]

[0024] The attention mechanism is used to calculate the weight of each historical query. This weight needs to be calculated based on the degree of correlation between the historical query and the current query. Take one type of weight as an example:

[0025]

[0026] Where d is the dimension of the hidden layer phase vector. As the weight of each query, the interaction scores obtained by each query are weighted and summed to obtain a comprehensive interaction score In a similar way, three other comprehensive interaction scores s can be obtained ad ,s rt ,s o :

[0027]

[0028] The document score calculation module combines the five scores s obtained previously rep ,s rm ,s ad ,s rt ,s o, and two additional fractions s ad and s f , these scores are spliced together through a fully connected layer to get the final document score, where s ad is the matching score between the current query and the candidate document calculated by the KNRM model, s f It is the score obtained by passing the relevant statistical features through the fully connected layer.

[0029] The query change modeling unit is used for two consecutive queries q i and q i-1 , calculate three word weights to describe the change relationship between the two queries, specifically,

[0030] First calculate q i The retention weight of the word in q i-1 The word and d i-1 The word representations in Then the query q is calculated i The attention weight of each word in compared to each word in sequence H:

[0031]

[0032] This weight is then normalized to obtain the word retention weight

[0033] In calculating q i When adding weights to words, the calculation method is the same as the retained weight, and the attention weight of each word is also calculated

[0034]

[0035] However, since the added word has a low similarity to the previous word, a minus sign is needed before normalization:

[0036] Calculate q i-1 When removing the weight of the word in the sentence, only q i-1 With q i So calculate q again i-1 The attention weight of each word in q i The association of words in Chinese:

[0037]

[0038] The normalization method is then the same as the method of adding weights to obtain the removal weight of the word:

[0039] The query change classifier divides query changes into four categories: generalization, exploitation, exploration, and new task. Generalization means that the next query is more general than the previous query, exploitation means that the next query is more specific than the previous query, exploration means that the next query discusses the same topic as the previous query but belongs to a different subtopic, and new task means that the next query discusses a completely different topic from the previous query. By comparing the information removed, added, and retained between the next query and the previous query, the query changes can be classified; the removed information representation of the previous query is obtained by comparing the removed representation with the original representation. Similarly, comparing the added and retained representations of the latter query with the original representation, we obtain These three representations are concatenated together and passed through a fully connected layer to obtain the predicted probability distribution of the four change categories.

[0040] The Attentive kernel pooling module is specifically as follows: after calculating a similarity matrix M between a query and a document word, the matching features f = {K1(M), ..., K k (M)}:

[0041]

[0042] where μ k and σ k is the kernel pooling parameter, β i,k It represents the weight of the i-th word in the query corresponding to the k-th core, which is calculated as follows: The entire process is denoted as f=Attentive_Kernel_Pooling(q, d, w).

[0043] The candidate document ranking model training process adopts a pairwise approach, and the loss function is designed as follows:

[0044]

[0045] Where R(d|q) is short for R(d|q,S), S contains all the sessions in the training set, Contains all clicked and unclicked documents corresponding to query q, where d + Indicates the clicked document, d - Represents unclicked documents; the query change classification task uses the cross entropy loss function commonly used in multi-classification problems:

[0046]

[0047] The loss functions of the two tasks are weighted summed to obtain an overall loss function.

[0048] The technical effects to be achieved by the present invention are:

[0049] We designed a model that improves conversational retrieval by modeling historical query changes. We assign weights to each query based on its evolution and leverage these weights for both representation-based matching and fine-grained word-level interaction. We also designed a query change classification task to enhance weight learning. Experimental results confirm the model's effectiveness.

[0050] Specifically, the technical effects are as follows:

[0051] (1) By analyzing historical query changes, three word weights are extracted to characterize query changes. The query change classification task is then used to improve the effect of weight learning.

[0052] (2) The obtained word weights are applied simultaneously to representation-based matching and word-level fine-grained interaction to improve the performance of conversational retrieval models. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The role of historical adjacent query change relations;

[0054] Figure 2 Overall framework diagram of the model; DETAILED DESCRIPTION

[0055] The following is a preferred embodiment of the present invention and combines the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to this embodiment.

[0056] The present invention proposes a conversational retrieval method that models the changes in historical adjacent queries.

[0057] Users often raise multiple queries in a query session, and after each query, they will choose a satisfactory document to click. For subsequent user queries, previous queries and clicked documents are often also of reference value. This information can tell the search engine what information the user is interested in. However, not all information is useful, and some information can even be misleading. For example Figure 1In the example above, the user's current query Q3 is about the cause of Voldemort's death, while in Q1 the user had already searched for the author of Harry Potter. Obviously, the user does not need to know the author's information when searching for Q3, so Q1 may be misleading to Q3. In order to distinguish which information is useful, many previous models have considered the analysis of query changes. However, previous models only considered the changes between the current query and historical queries, such as Figure 1 Arrows labeled 2 and 3 in the figure. These models ignore the evolving relationships between historical queries, as shown by arrow 1 in the figure. However, the evolving relationships between historical queries often contain useful information not included in the current query. As shown in the figure, if only the relationship between the current query and previous queries is considered, the search engine will assume that the word "Harry Potter" does not match the user's query intent because the user omitted it in the current query. However, the user likely omitted the word for simplicity. In fact, the user searched for content related to Harry Potter throughout the entire session. If the relationship between Q1 and Q2 is observed, and the user retained the word "Harry Potter" in Q2, it indicates that the word was likely the user's central topic of interest. Therefore, analyzing the evolving relationships between historical queries can help the model more accurately understand user intent, a feature previously ignored by models.

[0058] In order to model the changes in historical adjacent queries, the present invention proposes to calculate three types of word weights for any two adjacent queries: retain weight, remove weight, and add weight. These three word weights are used to weigh the user's preferred information and unwanted information. In order to better model the changes in historical queries, the present invention proposes to use a query change classification problem and designs a query change classifier (Reformulation Classifier) that takes the three word weights as input and outputs the change type. After analyzing the three word weights of historical query changes, these weights are used to perform word-level interaction and semantic-level matching between candidate documents and session history. Finally, a multi-task learning strategy is used to simultaneously learn the two tasks of candidate document ranking and query change classification.

[0059] In the conversational retrieval problem, each query may correspond to multiple click documents. In this invention, we consider one of the click documents that the user is satisfied with. Specifically, for the context S of the conversation, it contains historical queries and corresponding satisfied click documents S = [(q1, d1),…, (q t-1 ,d t-1 )]. Where q i represents the i-th query in the session, d i Indicates the user's satisfied click document corresponding to the i-th query. Also given the current query q t and candidate document d t, we need to calculate the relevance score R(d t |q t ,S).

[0060] Overall structure of the model

[0061] The overall structure of the model is Figure 2 The entire model can be divided into four parts: query term weighting, representation-based matching, term-based interaction, and document scoring. First, the term weighting module models the changing relationship between historical adjacent queries, calculates three term weights, and uses these to classify query changes. These three representations are then used to perform representation-based matching and term-based interaction between candidate documents and the conversation context, respectively, to obtain matching and interaction scores. Finally, a fully connected layer is used to calculate the document score.

[0062] Word weight calculation module

[0063] First, for each word in the query or document, Transformers is used to calculate the semantic representation of each word including the context:

[0064]

[0065]

[0066] where e q and e d are the initial word vectors of the words in the query and document, respectively, obtained through Fasttext training. p It is the position encoding, and Trm is the abbreviation of Transformers model.

[0067] For two adjacent historical queries q i-1 and q i , the change relationship between them is measured by three word weights, namely q i-1 The removal weight of each word in q i The added weight of each word in and retention weights The present invention designs a query change modeling unit (QRMU) to complete this process. In addition to the previous query and the next query, the input of QRMU also includes the user satisfaction click document corresponding to the previous query. Based on these three different types of word weights, three corresponding query representations can be obtained:

[0068] They are as well as Using the three query representations and the original word vector representations corresponding to the two queries as input, the query change classifier (RC) is used to predict the change type corresponding to this pair of adjacent queries:

[0069]

[0070] Representation-based matching module

[0071] After obtaining the three word weights, we first use a representation-based matching method to measure the degree of relevance between the candidate document and the conversation context. The above has introduced the method of obtaining three query representations through three word weights. In addition, we introduce the query representation obtained by the original word vector representation. For the above four query representations, Transformer is used to model the entire query. Specifically:

[0072]

[0073] The transformer can be used to interact with each other. Select the representation corresponding to the last query As the semantic representation of the entire query sequence. Similarly, the other three types of representations can be obtained By splicing these four representations together and passing them through a fully connected layer, we can obtain the semantic representation of the entire session. Compare this representation to the representation of the candidate document Calculate the similarity to get the matching score s based on the representation rep .

[0074] Word-level interaction module

[0075] To achieve fine-grained matching between historical queries and candidate documents, this paper designs an attentive kernel pooling based on kernel pooling, which uses three word weights to extract three types of interaction features: The structure of Attentive_Kernel_Pooling will be introduced later. Similarly, two other interactive features can be obtained through the other two word weights and In addition, ordinary kernel pooling can also be used to obtain unweighted interaction features. Passing these four types of matching features through the fully connected layer yields four types of interaction scores. To combine the interaction scores corresponding to each historical query, an attention mechanism is used to calculate the weight of each historical query. This weight is calculated based on the degree of correlation between the historical query and the current query. For example, one type of weight is:

[0076]

[0077] Where d is the dimension of the hidden layer phase vector. As the weight of each query, the interaction scores obtained for each query are weighted and summed to obtain a comprehensive interaction score s rm In a similar way, three other comprehensive interaction scores s can be obtained ad ,s rt ,s o .

[0078] Document score calculation module

[0079] In the final document score calculation module, the five scores s obtained previously are combined rep ,s rm ,s ad ,s rt ,s o , and two additional fractions s ad and s f , these scores are spliced together and passed through a fully connected layer to get the final document score. ad is the matching score between the current query and the candidate document calculated by the KNRM model, s f It is the score obtained by passing the relevant statistical features through the fully connected layer.

[0080] Query Change Modeling Unit (QRMU)

[0081] For two consecutive queries q i and q i-1 ,QRMU calculates three kinds of word weights to describe the change relationship between the two queries.

[0082] First calculate q i The retention weight of the word in q i-1 The word and d i-1 The word representations in Then the query q is calculated i The attention weight of each word in compared to each word in sequence H:

[0083]

[0084] This weight is then normalized to obtain the word retention weight

[0085] In calculating q i When adding weights to words, the calculation method is the same as the retained weight, and the attention weight of each word is also calculated However, since the added word has a low similarity to the previous word, a minus sign is needed before normalization:

[0086] In calculating q i-1 When removing the weight of the word in the sentence, only q i-1 With q i So calculate q again i-1 The attention weight of each word in q i The association of words in Chinese:

[0087]

[0088] The subsequent normalization is similar to the way weights are added, but a minus sign is added before the weight to get the word removal weight:

[0089] Query Change Classifier (RC)

[0090] This paper categorizes query changes into four categories: generalization, exploitation, exploration, and new task. Generalization means the next query is more general than the previous one, exploitation means it is more specific than the previous one, and exploration means the next query explores the same topic as the previous one, but belongs to a different subtopic. New task means the next query explores a completely different topic from the previous one. Query changes can be categorized by comparing the information removed, added, and retained between the previous and next queries.

[0091] In order to calculate the removed information, the removed representation of the previous query is compared with the original representation to obtain the removed information representation Similarly, comparing the added and retained representations of the latter query with the original representation, we obtain These three representations are concatenated together and passed through a fully connected layer to obtain the predicted probability distribution of the four change categories.

[0092] Attentive kernel pooling module

[0093] The kernel pooling module proposed in the KNRM model does not consider the weight of each word in the query. To incorporate word weights into kernel pooling, we improved the original algorithm.

[0094] In the original algorithm, after calculating a similarity matrix M between the query and the document word, the matching features f = {K1(M),…,K k (M)}:

[0095]

[0096] where μ k and σ k is the kernel pooling parameter used to measure different similarity distributions. In order to consider the weights of different words, the above formula is changed to:

[0097]

[0098] where β i,k It represents the weight of the i-th word in the query corresponding to the k-th core, which is calculated as follows:

[0099]

[0100] The whole process is recorded as f = Attentive_Kernel_Pooling(q,d,w)

[0101] Model training

[0102] The sorting model training process adopts a pairwise approach, and the loss function is designed as follows:

[0103]

[0104] Where R(d|q) is short for R(d|q,S). S contains all the sessions in the training set. Contains all clicked and unclicked documents corresponding to query q, where d + Indicates the clicked document, d - Indicates an unclicked document.

[0105] In the query change classification task, the cross entropy loss function commonly used in multi-classification problems is used:

[0106]

[0107] The loss functions of the two tasks are weighted and summed to obtain an overall loss function. All model parameters are updated through the back-propagation algorithm.

Claims

1. A conversational retrieval method that models changes in historical adjacent queries, characterized by: The method consists of four modules: a word weight calculation module, a representation-based matching module, a word-level interaction module, and a document score calculation module. The word weight calculation module models the change relationship of historical adjacent queries and calculates three word weights, namely, retention weight, removal weight, and addition weight. These three word weights are used to classify query changes. The results of the word weight calculation module are used in the representation-based matching module to perform representation-based matching between candidate documents and session contexts. At the same time, the word weights are also used in the word-level interaction module to perform fine-grained interaction at the word level, and representation-based matching and fine-grained interaction scores are obtained respectively. Subsequently, a fully connected layer of the document score calculation module is used to integrate five scores: representation-based matching score s rep , the interaction score s of the three word weights rm , three word weights and three comprehensive interaction scores ad ,s rt ,s o , and two additional fractions s ad and s f , these scores are spliced together through a fully connected layer to obtain the final document score R(d|q) for ranking, where s ad is the matching score between the current query and the candidate document calculated by the KNRM model, s f It is the score obtained by passing relevant statistical features through the fully connected layer; the document score is calculated for document ranking by search engines.

2. A conversational retrieval method for modeling historical adjacent query changes according to claim 1, wherein the method is used to solve the task of ranking candidate documents, characterized in that: The specific implementation of the word weight calculation module is as follows: for each word in a query or document, the semantic representation of each word including the context is calculated through Transformers: where e q and e d are the initial word vectors of the words in the query and document, respectively, obtained through Fasttext training, e p is the positional encoding, Trm is the Transformers model, For two adjacent historical queries q i-1 and q i , using the query change modeling unit to measure the change relationship between them through three word weights, namely q i-1 The removal weight of each word in q i The added weight of each word in and retain weights The query change modeling unit input includes not only the previous query and the next query, but also the user satisfaction click document corresponding to the previous query, that is, Based on these three different types of word weights, three corresponding query representations can be obtained, namely as well as These three query representations and the original word vector representations corresponding to the two queries are used as input, and the query change classifier is used to predict the change type corresponding to this pair of adjacent queries:

3. The conversational retrieval method for modeling historical adjacent query changes according to claim 2, characterized in that: After obtaining the three word weights, the representation-based matching module first uses a representation-based matching method to measure the degree of association between the candidate document and the conversation context. On the basis of the method of obtaining the three query representations based on the three word weights, the query representation obtained by the original word vector representation is additionally introduced. For the above four query representations, Transformer is used to model the entire query. Specifically: The representations of each query are interacted with each other through the transformer, and the representation corresponding to the last query is selected As the semantic representation of the entire query sequence, the other three types of representations can be obtained similarly By splicing these four representations together and passing them through a fully connected layer, we can obtain the semantic representation of the entire session. Compare this representation to the representation of the candidate document Calculate the similarity to get the matching score s based on the representation rep .

4. The conversational retrieval method for modeling historical adjacent query changes according to claim 3, characterized in that: The word-level interaction module designs an attentive kernel pooling, which uses three word weights to extract three types of interaction features: Similarly, two other interaction features can be obtained through the other two word weights and And use ordinary kernel pooling to obtain unweighted interaction features These four types of matching features are passed through the fully connected layer to obtain four types of interaction scores: The attention mechanism is used to calculate the weight of each historical query. This weight needs to be calculated based on the degree of correlation between the historical query and the current query. Take one type of weight as an example: Where d is the dimension of the hidden layer phase vector, As the weight of each query, the interaction scores obtained by each query are weighted and summed to obtain a comprehensive interaction score In a similar way, three other comprehensive interaction scores s can be obtained ad ,s rt ,s o :

5. The conversational retrieval method for modeling changes in historical adjacent queries according to claim 4, characterized in that: The query change modeling unit is used for two consecutive queries q i and q i-1 , calculate three word weights to describe the change relationship between the two queries, specifically, First calculate q i The retention weight of the word in q i-1 The word and d i-1 The word representations in Then calculate the query q i The attention weight of each word in compared to each word in sequence H: This weight is then normalized to obtain the word retention weight In calculating q i When adding weights to words, the calculation method is the same as the retained weight, and the attention weight of each word is also calculated However, since the added word has a low similarity to the previous word, a minus sign is needed before normalization: Calculate q i-1 When removing the weight of the word in the sentence, only q i-1 With q i So calculate q again i-1 The attention weight of each word in q i The association of words in For query q i-1 Indicates: The normalization is then done in the same way as the weight addition to obtain the word removal weight:

6. The conversational retrieval method for modeling historical adjacent query changes according to claim 5, characterized in that: The query change classifier divides query changes into four categories: generalization, exploitation, exploration, and new task, where generalization means that the latter query is more general than the former query, exploitation means that the latter query is more specific than the former query, exploration means that the latter query discusses the same topic as the former query but belongs to a different subtopic, and new task means that the latter query discusses a completely different topic from the former query. By comparing the information removed, added, and retained between the latter query and the former query, the query changes can be classified; the removed information representation of the former query is obtained by comparing the removed representation with the original representation. Similarly, comparing the added and retained representations of the latter query with the original representation, we obtain These three representations are concatenated together and passed through a fully connected layer to obtain the predicted probability distribution of the four change categories.

7. The conversational retrieval method for modeling changes in historical adjacent queries according to claim 6, characterized in that: The attentive kernel pooling is specifically as follows: after calculating a similarity matrix M between a query and a document word, the matching features f = {K1(M), ..., K k (M)}: where μ k and σ k is the kernel pooling parameter, β i,k It represents the weight of the i-th word in the query corresponding to the k-th core, which is calculated as follows: The entire process is denoted as f=Attentive_Kernel_Pooling(q, d, w).

8. The conversational retrieval method for modeling historical adjacent query changes according to claim 7, characterized in that: The candidate document ranking task training process adopts a pairwise approach, and the loss function is designed as follows: Where R(d|q) is short for R(d|q,S), S contains all the sessions in the training set, Contains all clicked and unclicked documents corresponding to query q, where d + Indicates the clicked document, d - represents the unclicked documents; the query change classification task uses the cross entropy loss function commonly used in multi-classification problems: The loss functions of the two tasks are weighted summed to obtain an overall loss function. The training process also includes using a multi-task learning strategy to simultaneously train the candidate document ranking task and the query change classification task.

Images