Tedious query thumbnail method fusing key concepts and potential concepts

Abstract

The invention discloses a lengthy query thumbnail method fusing key concepts and potential concepts, and relates to the field of natural language processing. Before information retrieval is carried out, lengthy query input is abbreviated. Key concepts representing query core content are combined with potential concepts that are important to understand the query but not explicitly expressed to generate a more complete and effective query. Specifically, firstly, a short and effective query is generated through fine tuning of a T5 model, and a short query serving as a key concept is generated according to semantic content of an input text; secondly, mining potential concepts of the original query by using a pseudo-correlation feedback method, and selecting n terms with the highest inverse document frequency in k documents ranked in the top in the initial retrieval result as the potential concepts of the original query; and finally, aggregating the two as a final query thumbnail result to realize lengthy query retrieval. According to the method, the recall rate and the accuracy of the retrieval result are improved.

Description

Technical Field

[0001] The invention discloses a method for abbreviating lengthy queries by integrating key concepts and potential concepts, and relates to the technical field of natural language processing. Background Art

[0002] Query shortening refers to reducing a lengthy query containing a large number of irrelevant terms into a concise and accurate form, and performing efficient retrieval based on the shortened query to improve the recall and accuracy of the retrieval results. In practical applications, the query shortening process usually involves identifying and removing stop words and unnecessary modifiers in the query, and extracting keywords and phrases that can accurately express the core content of the query.

[0003] There are two main methods in the study of query abbreviation tasks: traditional statistical methods and methods based on pre-trained models to extract keywords. However, the above methods will face information loss and limitations in semantic understanding when performing abbreviated queries. This is because (1) this method mainly abbreviates based on the importance score of the vocabulary, and the semantic incoherence in the abbreviated results destroys the overall semantic structure of the original query and cannot fully reflect the core content of the query. (2) It also ignores the potential concepts in the query. Potential concepts refer to the potential themes and contents behind the query statement, rather than the specific keywords or phrases directly revealed in the query. Due to the lack of supplementary potential concepts, the coverage of the retrieval results is insufficient. To address the above problems, this paper proposes a method for abbreviating lengthy queries that integrates key concepts and potential concepts. Summary of the invention

[0004] The present invention aims to provide a method for shortening lengthy queries by integrating key concepts and potential concepts, so as to improve the semantic understanding and core content extraction capabilities of lengthy queries, and at the same time supplement the potential concepts in the original queries, thereby improving the recall rate and accuracy of retrieval results.

[0005] The technical solution of the present invention is: a method for abbreviating lengthy queries by integrating key concepts and potential concepts, comprising:

[0006] For key concepts: Generate queries by fine-tuning the T5 model; generate a query as a key concept based on the semantic content of the input text;

[0007] For latent concepts: use the pseudo-relevance feedback method to mine the latent concepts of the original query; select the n terms with the highest inverse document frequency in the top k documents in the initial retrieval results as the latent concepts of the original query;

[0008] Finally, the query of key concepts and the potential concepts of the original query are aggregated to obtain the final abbreviated query.

[0009] The method specifically comprises the following steps:

[0010] Step 1. Download the Robust04 dataset, which contains 528,155 document data in the document set, 250 query data in the query set, and 311,410 relevance evaluations in the query relevance evaluation document;

[0011] Step 2: Convert the original query into a dense vector through the embedding layer and add the position code. Then, encode it through the 24-layer Transformer encoder to get the embedded representation. The decoder includes a masked multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network, and generates output through a 24-layer decoder. Finally, the output of the decoder passes through a linear layer and a Softmax layer to generate a probability distribution for each time step. Get the abbreviated query;

[0012] Step 3: Use the pseudo-relevance feedback method to mine important information that is not explicitly expressed in the query but is actually relevant for lengthy queries; select the n terms with the highest inverse document frequency in the top k documents in the initial retrieval results as the potential concepts of the original query;

[0013] Step 4: Aggregate the results of Step 2 and Step 3 to obtain the final abbreviated query. The final query is fed to the retriever for information retrieval.

[0014] As a preferred embodiment of the present invention, the Step 1 comprises:

[0015] Step 1.1. Download the public Robust2004 dataset from the Internet.

[0016] Step 1.2: Perform noise reduction on the document set in the Robust2004 dataset, delete blank lines, duplicate lines, and lines with illegal characters. <des>The queries under the label are used as the test set for the lengthy queries actually used. The training set is constructed manually, and the format corresponds to the query set <title>and< / title> <desc>Label.

[0017] As a preferred embodiment of the present invention, the Step 2 comprises:

[0018] Step 2.1. Input the original query Q = (q 1 ,q 2 ,…q |q| )(length |q|) is converted into a dense vector through the embedding layer and positional encoding is added to preserve the position information. Its embedding representation is E(Q)=(E(q 1 ),E(q 2 ),…,E(q |q| )).

[0019] Step 2.2, the input embedding E(Q) passes through a 24-layer Transformer encoder. Each layer contains two main components: a multi-head attention mechanism and a feedforward neural network. The output of the encoder is expressed as: H(Q) = (h 1 ,h 2 ,…,h |q| ).

[0020] Step 2.3, each layer of the decoder contains three main components: masked multi-head self-attention mechanism, encoder-decoder attention mechanism, and feedforward neural network. Decoding through the 24-layer decoder obtains the output representation of the decoder: Y = (y 1 ,y 2 ,…,y |m| ).

[0021] Step 2.4, the final output of the decoder is mapped to a vector of the vocabulary size through a linear layer, and the probability distribution of each time step is generated through a Softmax layer.

[0022] As a preferred embodiment of the present invention, the Step 3 comprises:

[0023] Step 3.1, given the original input query Q = (q 1 ,q 2 ,…q |q| ) and one of the documents in the document set D=(d 1 ,d 2 ,…,d |d| ). The lengths of the query and document are |q| and |d| respectively. The pre-trained BERT model is used to encode the query and document respectively. Each query and document is encoded into a high-dimensional vector representation. The vectors of the two are represented as E′(Q) and E(D) respectively.

[0024] Step 3.2, in the vector space of query and document, measure the similarity between the two by calculating the inner product of each word in the query and document. The element S of the similarity score matrix S is ij Represents the i-th word q in the query i and the jth word d in the document j The similarity.

[0025] Step 3.3: In order to obtain the final relevance score between the query and the document, this paper aggregates the similarity score matrix S through max-pooling.

[0026] Step 3.4: After the first intensive search, select the relevance score S corresponding to each query. q,d The top-k documents with the highest frequency (k=100 in the method of the present invention) are taken as the relevant document set. From these relevant document sets, the n terms with the highest inverse document frequency (n=6 in the method of the present invention) are extracted, and these n terms are the potential concepts.

[0027] Beneficial effects of the present invention:

[0028] The present invention proposes a method for abbreviating lengthy queries by integrating key concepts and potential concepts, combining key concepts that represent the core content of the query with potential concepts that are important for understanding the query but not explicitly expressed, thereby generating more complete and effective queries. The method mainly includes three aspects: first, using a pre-trained model to generate short and effective queries as key concepts, then using a pseudo-relevance feedback method to mine potential concepts from the relevant document set of the original query, and finally, aggregating the two as the final query abbreviated result to achieve lengthy query retrieval.

[0029] The lengthy query abbreviation method that integrates key concepts and potential concepts proposed in the present invention verifies the effectiveness of the method in the lengthy query abbreviation task and improves the recall rate and accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall framework diagram of a method for shortening a lengthy query by integrating key concepts and potential concepts provided by an example of the present invention;

[0031] Figure 2 A flow chart of a method for shortening a lengthy query by integrating key concepts and potential concepts provided by an example of the present invention;

[0032] Figure 3 The example of using AdaptKeyBERT to perform query abbreviation and retrieval is provided in the example of the present invention. DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the embodiment and the present invention will be described in detail below with reference to the accompanying drawings.

[0034] Example 1, a method for abbreviating lengthy queries that integrates key concepts and potential concepts, the method is divided into two parallel steps. (1) For key concepts: generate short and effective queries by fine-tuning the T5 model. This is more of a generative method. Instead of simply extracting core terms from the input text, it generates a short query as a key concept based on the semantic content of the input text. (2) For potential concepts: use the pseudo-correlation feedback method to mine the potential concepts of the original query; select the n terms with the highest inverse document frequency in the top k documents in the initial retrieval results as the potential concepts of the original query. Finally, the results obtained in the two steps are aggregated to obtain the final abbreviated query.

[0035] The present invention is described below in conjunction with specific embodiments, and the method specifically comprises: Figure 1 , Figure 2 , Figure 3 As shown;

[0036] Step 1. Download the Robust04 dataset. This dataset is mainly used to verify the robustness of search engines. It contains some difficult queries in information retrieval systems and aims to test the performance of retrieval systems when facing challenging queries. The document set contains 528,155 document data, the query set contains 250 query data, and the query relevance evaluation document contains 311,410 relevance evaluations.

[0037] Step 2: Convert the original query into a dense vector through the embedding layer and add position encoding. Then, it is encoded through a 24-layer Transformer encoder to obtain an embedded representation. The decoder includes a masked multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network, and generates output through a 24-layer decoder. Finally, the output of the decoder passes through a linear layer and a Softmax layer to generate a probability distribution for each time step. The abbreviated query is obtained.

[0038] Step 3: Use pseudo-relevance feedback method to mine important information that is not explicitly expressed in the query but is actually relevant for lengthy queries. Select the n terms with the highest inverse document frequency in the top k documents in the initial retrieval results as the potential concepts of the original query.

[0039] Step 4: Finally, aggregate the results of the two steps to obtain the final abbreviated query. The final query is fed into the retriever for information retrieval.

[0040] As a preferred embodiment of the present invention, the Step 1 comprises:

[0041] Step 1.1. Download the public Robust2004 dataset from the Internet.

[0042] Step 1.2: Perform noise reduction on the document set in the Robust2004 dataset, delete blank lines, duplicate lines, and lines with illegal characters. <des>The queries under the label are used as the test set for the lengthy queries actually used. The training set is constructed manually, and the format corresponds to the query set <title> and< / title> <desc>Label.

[0043] As a preferred embodiment of the present invention, the Step 2 comprises:

[0044] Step 2.1. Convert the original input query Q = (q1, q2, ... q|q|) (length |q|) into a dense vector through the embedding layer and add position encoding to retain the position information. The specific formula is as follows:

[0045]

[0046] Its embedding is expressed as: E(Q) = (E(q 1 ),E(q 2 ),…,E(q |q| )).

[0047] Step 2.2, the input embedding E(Q) passes through a 24-layer Transformer encoder. Each layer contains two main components: a multi-head attention mechanism and a feedforward neural network. The specific formula is as follows:

[0048]

[0049] The final encoder output is expressed as: H(Q) = (h 1 ,h 2 ,…,h |q| ).

[0050] Step 2.3, each layer of the decoder contains three main components: masked multi-head self-attention mechanism, encoder-decoder attention mechanism, and feedforward neural network. Decoding is performed through a 24-layer decoder. The decoding process formula is as follows:

[0051]

[0052] The output of the decoder is: Y = (y 1 ,y 2 ,…,y |m| ).

[0053] Step 2.4, the final output of the decoder is mapped to a vector of the vocabulary size through a linear layer, and the probability distribution of each time step is generated through a Softmax layer. The specific formula is as follows:

[0054] P(y i )=Softmax(W 0 ·y i +b 0 )

[0055] Where W 0 is the weight parameter, and b 0 is bias.

[0056] As a preferred embodiment of the present invention, the Step 3 comprises:

[0057] Step 3.1, given the original input query Q = (q 1 ,q 2 ,…q |q| ) and one of the documents in the document set D=(d 1 ,d 2 ,…,d |d| ). The lengths of the query and document are |q| and |d| respectively. The pre-trained BERT model is used to encode the query and document respectively. Each query and document will be encoded into a high-dimensional vector representation. The vectors of the two are represented as E′(Q) and E(D) respectively. The encoding formula is as follows:

[0058] E ′ (Q)=BERT(Q)

[0059] E(D)=BERT(D)

[0060] Step 3.2, in the vector space of query and document, measure the similarity between the two by calculating the inner product of each word in the query and document. The element S of the similarity score matrix S is ij Represents the i-th word q in the query i and the jth word d in the document j The specific formula is as follows:

[0061]

[0062] Step 3.3: In order to obtain the final relevance score between the query and the document, this paper aggregates the similarity score matrix S through max-pooling. The specific formula is as follows:

[0063]

[0064] Step 3.4: After the first intensive search, select the relevance score S corresponding to each query. q,d The top-k documents with the highest frequency (k=100 in the method of the present invention) are taken as the relevant document set. From these relevant document sets, the n terms with the highest inverse document frequency (n=6 in the method of the present invention) are extracted, and these n terms are the potential concepts.

[0065] In order to compare with the existing methods, the present invention uses the public Robust2004 dataset, where the number of query pairs in the training set is 1000 and the number of queries in the test set is 250. There are 528155 document data in the document set. The selected existing comparison methods include: original method, Yake, AdaptKeyBERT, TopicRank, RM3. The original query is the query pair of Robust2004. <desc>query under the tag. Yake is a completely statistical method that has been recently introduced for keyword extraction. It relies on capturing five statistical features of the terms in the document and integrating these feature scores into the final score for key phrase determination. AdaptKeybert is a keyword extraction technique based on BERT, proposed by AmanPriyanshu et al. in 2022. The technique aims to automatically extract the most similar keywords and key phrases to the input content. AdaptKeybert uses the powerful semantic understanding ability of the BERT model to better understand and extract the semantic information of the text, thereby providing more accurate and relevant keyword extraction results. TopicRank is another graph-based keyword extraction algorithm. In TopicRank, graph nodes are not just single words. Instead, documents are represented as a complete graph of topics and relationships between these topics. Topics are defined as a set of similar words and multi-word phrases. RM3 (Relevance Model with Pseudo Rel-evance Feedback) is a classic feedback algorithm for information retrieval. The algorithm dynamically adjusts the query model by leveraging the retrieved relevant documents to improve the quality and accuracy of retrieval results. In RM3, an initial query is first used to retrieve a batch of relevant documents from the document collection, and then the query is remodeled based on the information in these feedback documents to improve the original query.

[0066] The results of the compared methods and the method of the present invention on the Robust2004 dataset are shown in Table 1 below, and the best performing part of each indicator is highlighted in bold:

[0067] Table 1 Comparison of experimental results of different models (BM25 retrieval model)

[0068]

[0069]

[0070] Table 2 Comparison of experimental results of Yake, AdaptKeyBERT, TopicRank, and Ours (ColBERT retrieval model)

[0071]

[0072] As shown in Table 1 and Table 2. The experimental results show that: 1) The corresponding model of the present invention failed to achieve the best MAP index under the BM25 retrieval model, but it was better than the baseline model in other indicators of the BM25 model and all indicators of the ColBERT model. This proves that the method proposed in the present invention is effective in both sparse retrieval models and dense retrieval models. 2) Compared with the methods that only extract key concepts (TopicRank, Yake, AdaptKeyBERT), the method proposed in this paper is better than all the methods that only extract key concepts. The experimental results prove that the method that only extracts key concepts can use potential concepts as a supplement to improve the quality of queries. 3) Compared with the method that only mines potential concepts (RM3), the method proposed in this invention is more effective. The experimental results prove that the method that mines potential concepts can use the key concepts of the original query to generate a more complete query.

[0073] like Figure 3 As shown in the figure, when AdaptKeyBERT is used to abbreviate the original query: What adverse effects have people experienced while taking aspirin repeatedly?, the abbreviated query is: aspirin, effects, taking (aspirin, influence, taking), which does not fully capture the core content of the query, and does not include "adverse" and "repeatedly", which are important terms in the original query. In addition, the abbreviated query contains some incoherent keywords. Figure 3 Compared with the standard related documents in the , the returned documents are less relevant.

[0074] In order to verify the effectiveness of the two modules proposed in this paper, this paper uses the Robust2004 dataset to design ablation experiments under the dense retrieval model (ColBERT) on four types of evaluation indicators. 1) W / O key_concept: remove the key concept generation module based on T5-Large; 2) W / O latent_concept: remove the latent concept mining module based on ColBERT. The specific ablation experiment results are shown in Table 3:

[0075] Table 3 Ablation experiments conducted under the dense retrieval model (ColBERT)

[0076]

[0077] The experimental results show that after removing any of the two modules, the model performance decreases in four categories of indicators, which shows that the two modules proposed in the present invention have a certain degree of contribution to the performance of the task of shortening lengthy queries. 1) Removing the key concept generation module based on T5-Large, compared with the complete model, all four indicators have decreased. This shows that if there are only latent concepts without the original query content and semantics, the query is not clear enough. The relevance and accuracy of the retrieval results are reduced. 2) Removing the latent concept mining module based on ColBERT, compared with the complete model, all four indicators have decreased. This shows that the lack of latent concepts will miss important background information, which limits the retrieval system in understanding the query. The recall rate of the retrieval results is reduced.

[0078] Through the above ablation experiment comparison and analysis, the effectiveness of the method of the present invention for different components of the model is verified.

[0079] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.< / desc> < / desc> < / des> < / desc> < / des>

Claims

1. A method for shortening lengthy queries by integrating key concepts and potential concepts, characterized in that: include: For key concepts: Generate queries by fine-tuning the T5 model; generate a query as a key concept based on the semantic content of the input text; For latent concepts: use the pseudo-relevance feedback method to mine the latent concepts of the original query; select the n terms with the highest inverse document frequency in the top k documents in the initial retrieval results as the latent concepts of the original query; Finally, the query of key concepts and the potential concepts of the original query are aggregated to obtain the final abbreviated query.

2. The method for shortening a lengthy query by integrating key concepts and potential concepts as claimed in claim 1, characterized in that: The following steps are involved: Step 1, obtain the document set, query set and query relevance evaluation document; Step 2: The original query is converted into a dense vector through the embedding layer and positional encoding is added; then, it is encoded through a 24-layer Transformer encoder to obtain an embedded representation; The decoder includes a masked multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network, and generates output through a 24-layer decoder; finally, the output of the decoder passes through a linear layer and a Softmax layer to generate a probability distribution for each time step to obtain the abbreviated query. Step 3: Use the pseudo-relevance feedback method to mine important information that is not explicitly expressed in the query but is actually relevant for lengthy queries; select the n terms with the highest inverse document frequency in the top k documents in the initial retrieval results as the potential concepts of the original query; Step 4: Aggregate the results of Step 2 and Step 3 to obtain the final abbreviated query, and feed the final query to the retriever for information retrieval.

3. The method for shortening a lengthy query by integrating key concepts and potential concepts as claimed in claim 2, characterized in that: The specific implementation of Step 1 includes: Step1.1, download the public Robust2004 dataset from the Internet; Step 1.2: Perform noise reduction on the document set in the Robust2004 dataset, delete blank lines, duplicate lines, and lines with illegal characters; extract the query set in the Robust2004 dataset. <des>The queries under the label are used as the test set of lengthy queries actually used; the training set is constructed manually, and the format corresponds to the query set <title> and< / title> <desc> Label.< / desc> < / des> 4. The method for shortening a lengthy query by integrating key concepts and potential concepts as claimed in claim 2, characterized in that: The specific implementation of Step 2 includes: Step 2.

1. Input the original query Q = (q1, q2, ...q |q| ), with a length of |q|, is converted into a dense vector through an embedding layer and positionally encoded to preserve position information; its embedding representation is E(Q)=(E(q1),e(q2),…,E(q |q| )); Step 2.2: The input embedding representation E(Q) passes through a 24-layer Transformer encoder; each layer contains two main components: a multi-head attention mechanism and a feedforward neural network. The output of the encoder is represented as: H(Q) = (h1, h2, …, h |q| ); Step 2.3, each layer of the decoder contains three main components: masked multi-head self-attention mechanism, encoder-decoder attention mechanism, feedforward neural network; decoded by a 24-layer decoder, the output representation of the decoder is: Y = (y1, y2, ..., y |m| ); Step 2.4, the final output of the decoder is mapped to a vector of the vocabulary size through a linear layer, and the probability distribution of each time step is generated through a Softmax layer.

5. A method for shortening a lengthy query by integrating key concepts and potential concepts as claimed in claim 2, characterized in that: The specific implementation of Step 3 includes: Step 3.1, given the original input query Q = (q1, q2, ...q |q| ) and one of the documents in the document set D=(d1,d2,…,d |d| ); the lengths of the query and document are |q| and |d| respectively; the pre-trained BERT model is used to encode the query and document respectively, and each query and document is encoded into a high-dimensional vector representation, and the vectors of the two are represented as E′(Q) and E(D) respectively; Step 3.2: In the vector space of the query and document, the similarity between the query and the document is measured by calculating the inner product of each word in the query and the document; the element S of the similarity score matrix S is ij Represents the i-th word q in the query i and the jth word d in the document j similarity; Step 3.3, to obtain the final relevance score of the query and document, the similarity score matrix S is aggregated through maximum pooling; Step 3.4: After the first intensive search, select the relevance score S corresponding to each query. q,d The top-k documents with the highest inverse document frequency are taken as relevant document sets, and the n terms with the highest inverse document frequency are extracted from these relevant document sets. These n terms are the potential concepts.

Images