Query expansion method and system based on visual semantic similarity

Abstract

The present invention provides a query expansion method and system based on visual semantic similarity, comprising: retrieving user-submitted query information to obtain retrieved documents; obtaining user visual attention information when the user browses the retrieved documents, extracting user-focused words of interest from the user visual attention information at the term level to form a visual interest word set, and calculating the weight of each interest word; training a semantic model, using the trained semantic model to calculate the semantic similarity between each word in the visual interest word set and the original query, screening out words with a similarity not less than a minimum threshold to form a candidate expansion word set; recalculating the weight of each expansion word in the candidate expansion word set, selecting the top k expansion words based on the weights, reconstructing them with the original query information, and outputting new query information. The present invention takes into account the personalized needs of the current user during retrieval, significantly improving retrieval accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval, and in particular relates to a query expansion method and system based on visual semantic similarity. Background Art

[0002] Search engines provide an input box as an input interface, allowing users to enter their query. The retrieval system matches the user's input keywords with keywords in the index and returns documents containing the corresponding keywords. With the increasing maturity of search engine technology, retrieval efficiency has reached the point where relevant web pages can be returned in real time. However, retrieval accuracy still needs to be improved. Too short queries and ambiguous terms prevent users from accurately expressing their search intent, resulting in a large number of irrelevant web pages returned in the search results, requiring users to spend a considerable amount of time browsing to find the information they need. To address this issue, query expansion technology has emerged. Current mainstream query expansion methods share a common shortcoming: their processing only caters to the search preferences of the majority of users and lacks personalized understanding of the current user. Query expansion based on user query logs or user click models can, to some extent, address this shortcoming. However, past log information cannot fully represent the user's current query intent. If the user's query interests change, this method cannot capture the changed information needs in real time, which can also reduce the accuracy of the search results. Therefore, developing a query expansion method and model based on visual semantic similarity to effectively overcome the defects in the above-mentioned related technologies has become a technical problem that needs to be urgently solved in the industry. Summary of the Invention

[0003] One purpose of the present invention is to address the deficiencies of the prior art and provide a query expansion method based on visual semantic similarity, which takes into account the personalized needs of the current user during retrieval and greatly improves the accuracy of retrieval.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A query expansion method based on visual semantic similarity includes the following steps:

[0006] Step 1: Search the query information submitted by the user to obtain the retrieved documents;

[0007] Step 2: When the user browses the retrieved document, the user's visual attention information is obtained, and the user's interest words are extracted from the user's visual attention information at the term level to form a visual interest word set and the weight of each interest word is calculated;

[0008] Step 3: Train the semantic model, use the trained semantic model to calculate the semantic similarity between each word in the visual interest word set and the original query, and select words with a similarity not less than a minimum threshold to form a candidate expansion word set;

[0009] Step 4: Recalculate the comprehensive weight of each expansion word in the candidate expansion word set based on the weight calculated in step 2 and the semantic similarity obtained in step 3, and then determine the weight ratio of the expansion word based on the comprehensive weight. Select the top k expansion words and add them to the original query information according to the weight ratio for reconstruction, and output the new query information.

[0010] Furthermore, the query information is input into the search engine to retrieve documents, and the BM25F scoring algorithm is used to calculate the similarity score between the query term and the document. The documents are sorted from high to low according to the final score, and the top N documents with the highest scores are displayed to the user.

[0011] Furthermore, in step 2, an eye tracker or other visual tracking device is used to capture the user's visual gaze data information, and the information is converted into the interest area of the word through OCR or eye movement processing software, and the user's visual interest word set is extracted from the interest area of the word.

[0012] Furthermore, the initial weight of interest words is determined as follows:

[0013] First, normalize the indicators of each interest word according to the following formula:

[0014]

[0015] A certain weight ratio is assigned to a data indicator selected from the normalized indicators And consider the word frequency ζ in the document t , get the comprehensive weight of the visual interest word:

[0016]

[0017] Furthermore, in step 3, the semantic model is trained using two datasets, Sogou-QCL and Wikipedia.

[0018] Furthermore, the candidate expansion word set W eye-ds The construction method is:

[0019] Calculate the visual interest word set W eye Interest words in Calculate the similarity with the query words in the original query Q in turn, and take the one with the largest similarity value as the interest word Similarity with the original query Q

[0020]

[0021] According to the preset minimum similarity threshold minqtcos, W eye The interest terms that are not less than minqtcos are included in the candidate expansion word set W eye-ds .

[0022] Furthermore, in step 4, the candidate expansion word set W eye-ds The method for recalculating the weight value of each candidate word in is:

[0023] The weight of the expanded word calculated according to step 2 and the similarity value with the original query calculated in step 3 Calculate the weight of the candidate expansion word At the same time, consider the word frequency ζ in the document t , get the comprehensive weight of the word:

[0024]

[0025]

[0026] Another object of the present invention is to provide a system according to the above-mentioned query expansion method based on visual semantic similarity, comprising:

[0027] The original query processing module is used to process the original query information input by the user in the interactive interface into an original query vector and input it into the search engine for retrieval to obtain the retrieved document;

[0028] An eye movement data processing module is used to obtain and process the visual attention information generated by the user when browsing the retrieved documents, and generate a set of visual interest words for the user;

[0029] A semantic similarity calculation module is used to calculate the similarity between the original query information and the words in the visual interest word set, and select words with a similarity not less than a minimum threshold to form a candidate expansion word set;

[0030] The query reconstruction module is used to recalculate the weight of each expansion word in the candidate expansion word set, select the first k expansion words according to the weight, reconstruct them with the original query information, and then output the new query information.

[0031] Compared with existing technologies, the present invention offers the following advantages: It proposes a real-time, personalized query expansion method. By capturing users' browsing behavior on search result documents, it can extract keywords of interest for query expansion, enabling the reconstructed query to reflect the user's current information needs. This method not only overcomes the drawback of traditional query expansion models, which employ a "universal" expansion model for all users and fail to reflect their personalized needs, but also addresses the problem of query log-based query expansion techniques failing to meet time-sensitive query requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a query expansion method based on visual semantic similarity provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the practical principle of a query expansion method based on visual semantic similarity provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the structure of a query expansion system based on visual semantic similarity provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0037] The present invention will be further described below with reference to specific examples, but they are not intended to limit the present invention.

[0038] like Figure 1 As shown, an embodiment of the present invention provides a query expansion method based on visual semantic similarity, comprising the following steps:

[0039] Step 1: Search the query information submitted by the user to obtain the retrieved documents;

[0040] See Figure 2In this step, the user enters the query information in the interactive interface, and the query parser is used to segment the query words, remove stop words, and obtain the processed query words. The tools used here can be the ChineseAnalyzer analyzer and the "Harbin Institute of Technology Stop Word List" in the Jieba tool. The processed query words are then quantified into query vectors using the vector space model and input into the search engine for search. When searching, you can specify which "Fields" to search in and set the weight of each Field to obtain a series of documents. The BM25F scoring model is then used to calculate the similarity score between the query word and the document:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] in, It is BM25 Score(Q,d) The first part is the BIM calculation formula. The BIM formula means that for feature words that appear in both Q and d, the estimated values of all feature words are accumulated to obtain the similarity score between the query Q and the document d. N is the total number of documents, R is the number of relevant documents, NR is the number of irrelevant documents, K factor represents the consideration of document length, b is the document length adjustment factor, if b is set to 0, the influence of document length factor is not considered. k1 and k2 are word frequency adjustment factors, n i Indicates that it contains feature d i The number of documents, r i Indicates the number of relevant documents, f i Indicates the frequency of the feature word in document d, qf i Represents the frequency of the feature word in the query, d l Indicates the length of document d, avgd l Indicates the average length of all documents. The difference between BM25F and BM25 is that f i ′. For a document d, it can be divided into u domains, and the weight of each part is w k , f ui Indicates the frequency of the feature word appearing in domain u, B u Indicates the length factors of the different parts.

[0047] The documents are sorted from high to low based on the final scores, and the top N documents with the highest scores are displayed to the user. In this embodiment, the relevant parameters of the BM25F model include two hyperparameters: k1, which adjusts the frequency of the query term in the document, and b, which adjusts the document length. Based on past experience, k1 is generally set to 1.2 and b is set to 0.75.

[0048] Step 2: When the user browses the retrieved document, the user's visual attention information is obtained, and the user's interest words are extracted from the user's visual attention information at the term level to form a visual interest word set and the weight of each interest word is calculated;

[0049] This embodiment uses an eye tracker or other visual tracking device to capture the user's visual attention information, wherein the visual attention information generated during the user's search and browsing process extracts the user's visual interest word set There are two steps involved:

[0050] 1) Acquisition of words of interest: The data information based on the user's gaze point captured by an eye tracker or other visual tracking device is converted into data information based on keywords through certain technologies. This process can be achieved through OCR technology, or it can be achieved by directly dividing the area of interest using eye movement processing software. For example, the corresponding AOI Tag (area of interest tag) can be created through the "AOI Tool" in Tobbi Pro Lab to convert it into the area of interest of the word; the "Automatically generate area of interest (AOI)" function provided in Tobbi Pro Lab is used to divide the area of interest based on single words, and the "AOI Tool" is used to manually set the corresponding word AOI tag on the Tobbi Pro Lab software based on the jieba word segmentation results to obtain the word of interest. The Tobbi Pro Lab software can export various indicators of the corresponding word of interest, including total gaze time, average gaze time, number of gazes, first gaze duration, and average pupil diameter;

[0051] 2) Determination of the initial weight of interest words: Normalize the above five indicators of each interest word according to the following formula:

[0052]

[0053] in represents the normalized value of the kth sample of the ith feature (index), λ represents a constant used to map each value to the range (0,1), max(x i ) and min(x i ) represent the maximum and minimum values of the i-th feature respectively.

[0054] After normalizing each indicator, select one of the indicators and assign it a certain weight ratio. And consider the word frequency ζ in the document t , get the initial weight of the visual interest word:

[0055]

[0056] Step 3: Train the semantic model, use the trained semantic model to calculate the semantic similarity between each word in the visual interest word set and the original query information, and select words with a similarity not less than a minimum threshold to form a candidate expansion word set;

[0057] In the embodiment of the present invention, the semantic model is first trained, and then the trained semantic model is used to calculate the semantic similarity between each word in the visual interest word set and the original query. In this embodiment, the semantic model is trained using two data sets, Sogou-QCL and Wikipedia. Specifically, the word vector is trained using the Skip-gram model of Word2vec, and the parameters set during training are as follows: the dimension size of the output word vector is set to 100, the window size is set to 10, and the training algorithm is set to negative sampling technology; the min_count parameter is used to filter words, and words with a frequency less than min_count will be ignored. Considering that the data set selected by Sogou-QCL is not large, the Model qcl The min_count is set to 0. The Wikipedia Chinese dataset has a large amount of data, so the min_count is set to 5.

[0058] According to the trained semantic model, the similarity between the interest word and the original query is calculated, and the candidate expansion word set W that filters out the noise terms is output. eye-ds , the specific method is as follows:

[0059] W eye Interest words in Calculate the similarity with the query words in the original query information Q in turn, and take the one with the largest similarity value as the interest word Similarity with the original query Q

[0060]

[0061] in Interest words With the i-th query word q i The cosine similarity between .

[0062] According to the preset minimum similarity threshold minqtcos, where the minimum similarity threshold minqtcos is determined by the actual effect of retrieval in the instance, W eye The interest terms that are not less than minqtcos are included in the candidate expansion word set W eye-ds .

[0063] Step 4: Recalculate the comprehensive weight of each expansion word in the candidate expansion word set based on the weight calculated in step 2 and the semantic similarity obtained in step 4, then determine the weight ratio of the expansion words based on the comprehensive weight, select the top k expansion words and add them to the original query information according to the weight ratio to reconstruct the query information, and output the new query information;

[0064] For the candidate expansion word set W eye-ds The candidate words in need to recalculate the weight value. According to the weight value calculated based on the eye movement data in step 2 and similarity value with the original query Calculate the weight of the candidate expansion word At the same time, consider the word frequency ζ in the document t , get the comprehensive weight of the word:

[0065]

[0066]

[0067] In W eye-ds The first k terms are selected as expanded terms, and they are added to the original query according to the weight ratio to reconstruct the new query formula for secondary retrieval, wherein the weight ratio of the expanded query term is the product of the comprehensive weight of the expanded query term and its reconstruction ratio. According to subsequent experiments, the parameter values with the best retrieval effect are determined: the reconstruction ratios of the original query and the expanded terms are 1.0 and 0.8 respectively. The present invention also discusses the impact of the value of the number of expanded terms k on the retrieval performance, and finds that the retrieval performance gets better and better with the increase of the number of expanded terms. When k=13, the average accuracy (MAP) and normalized discounted cumulative gain (NDCG) are the highest. When the number exceeds 13, the query performance begins to decline.

[0068] In summary, the present invention utilizes the user's visual attention information to design an effective query expansion model - a query expansion model based on visual semantic similarity. The retrieval system uses the visual attention mechanism to automatically capture the keywords that the user is interested in, and after the process of term screening, weight calculation, etc., expands the keywords that represent the user's needs to the query for feedback. Visual attention information belongs to the category of implicit feedback. It not only does not cause an interactive burden on the user's retrieval, but also the interest words extracted at the term level will be closer to the user's query needs. The detailed workflow of the model is: through the collection and data analysis of the user's visual attention information, the keywords that the user is more concerned about are extracted at the term level to form a visual interest word set and the weight of each interest word is calculated. Then, the trained semantic model is used to calculate the semantic similarity between each word in the visual interest word set and the original query, and words with a similarity threshold of not less than the minimum similarity threshold are screened out to form a candidate expansion word set, and the weight of each expansion word is recalculated. Finally, the top k expansion words are selected and added to the original query for re-search to return a more accurate retrieval result to the user.

[0069] like Figure 3 As shown, an embodiment of the present invention further provides a system according to the above-mentioned query expansion method based on visual semantic similarity, comprising:

[0070] The original query processing module is used to process the original query information input by the user in the interactive interface into an original query vector and input it into the search engine for retrieval to obtain the retrieved document;

[0071] An eye movement data processing module is used to obtain and process the visual attention information generated by the user when browsing the retrieved documents, and generate a set of visual interest words for the user;

[0072] A semantic similarity calculation module is used to calculate the similarity between the original query information and the words in the visual interest word set, and select words with a similarity not less than a minimum threshold to form a candidate expansion word set;

[0073] The query reconstruction module is used to recalculate the weight of each expansion word in the candidate expansion word set, select the first k expansion words according to the weight, reconstruct them with the original query information, and then output the new query information.

[0074] The above are only preferred embodiments of the present invention and do not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the present invention specification should be included in the protection scope of the present invention.

Claims

1. A query expansion method based on visual semantic similarity, characterized in that: The steps include: Step 1: Search the query information submitted by the user to obtain the retrieved documents; Step 2: When the user browses the retrieved document, the user's visual attention information is obtained, and the user's interest words are extracted from the user's visual attention information at the term level to form a visual interest word set and the weight of each interest word is calculated; Step 3: Train the semantic model, use the trained semantic model to calculate the semantic similarity between each word in the visual interest word set and the original query, and select words with a similarity not less than a minimum threshold to form a candidate expansion word set; Step 4: Recalculate the comprehensive weight of each expansion word in the candidate expansion word set based on the weight calculated in step 2 and the semantic similarity obtained in step 3, and then determine the weight ratio of the expansion word based on the comprehensive weight. Select the top k expansion words and add them to the original query information according to the weight ratio for reconstruction, and output the new query information.

2. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: The query information is input into the search engine to retrieve documents. The BM25F scoring algorithm is used to calculate the similarity score between the query term and the document. The documents are sorted from high to low according to the final score, and the top N documents with the highest scores are displayed to the user.

3. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: In step 2, an eye tracker or other visual tracking device is used to capture the user's visual gaze data information, and the information is converted into the interest area of the word through OCR or eye movement processing software, and the user's visual interest word set is extracted from the interest area of the word.

4. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: Determination of the initial weight of interest words: First, normalize the indicators of each interest word according to the following formula: A certain weight ratio is assigned to a data indicator selected from the normalized indicators And consider the word frequency ζ in the document t , get the comprehensive weight of the visual interest word:

5. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: In step 3, the semantic model is trained using the Sogou-QCL and Wikipedia datasets.

6. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: Candidate expansion word set W eye-ds The construction method is: Calculate the visual interest word set W eye Interest words in Calculate the similarity with the query words in the original query Q in turn, and take the one with the largest similarity value as the interest word Similarity with the original query Q According to the preset minimum similarity threshold minqtcos, W eye The interest terms that are not less than minqtcos are included in the candidate expansion word set W eye-ds .

7. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: In step 4, the candidate expansion word set W eye-ds The method for recalculating the weight value of each candidate word in is: The weight of the expanded word calculated according to step 2 and the similarity value with the original query calculated in step 3 Calculate the weight of the candidate expansion word At the same time, consider the word frequency ζ in the document t , get the comprehensive weight of the word:

8. The query expansion method based on visual semantic similarity according to claim 1, characterized in that: After obtaining the comprehensive weight, the reconstruction ratio is set, and the comprehensive weight of the expansion word is multiplied by the reconstruction ratio to obtain the weight ratio of the expansion word.

9. A system according to the query expansion method based on visual semantic similarity according to any one of claims 1 to 8, characterized in that: include: The original query processing module is used to process the original query information input by the user in the interactive interface into an original query vector and input it into the search engine for retrieval to obtain the retrieved document; An eye movement data processing module is used to obtain and process the visual attention information generated by the user when browsing the retrieved documents, and generate a set of visual interest words for the user; A semantic similarity calculation module is used to calculate the similarity between the original query information and the words in the visual interest word set, and select words with a similarity not less than a minimum threshold to form a candidate expansion word set; The query reconstruction module is used to recalculate the weight of each expansion word in the candidate expansion word set, select the first k expansion words and reconstruct them with the original query information, and then output the new query information.

Images