A Short Text Query Expansion and Retrieval Method Based on Word Vector

Abstract

A short text query expansion and retrieval method based on word vectors, specifically including: A, short text corpus information preprocessing; B, training model to represent each word in the corpus dictionary with word vectors; C, query expansion; D .Using the query expansion word set and BM25 retrieval model to obtain text candidate sets; E. Topic extraction of short texts; F. Calculation of text vectors of short texts; G. Reordering of short texts returned by traditional retrieval models. The invention can more accurately and effectively meet the user's retrieval requirements, and the query expansion module can find words that can express the user's intention according to the existing data to perform query expansion.

Description

technical field [0001] The invention relates to the technical fields of data mining and search engines, in particular to a short text query expansion and retrieval method based on word vectors. Background technique [0002] With the rapid development of computer and Internet, it becomes more and more difficult to accurately obtain information from massive information resources. A large part of the massive information exists in the form of short text, and short text is also an indispensable data form in people's daily life. Short text information mainly includes blog messages, microblog information, short messages, chat records, etc., and is characterized by short message length, flexible language form, huge data scale, strong timeliness, and fast update speed. Traditional search engines are not very accurate in these short text retrievals, and cannot meet people's needs for accurate information acquisition. Therefore, the present invention designs and implements a search en...

AI Technical Summary

Problems solved by technology

The disadvantage of this type of method is: when the user gives a search term, the search engine can only return documents containing the search term, but cannot return other documents that are semantically related but expressed in different words

The disadvantage of this type of method is: when the user gives a search term, the search engine will introduce a lot of noise information, although the recall rate of the retrieval system is improved to a certain extent, but it also introduces a large amount of irrelevant text, reducing the search accuracy

[0016] These methods only enrich the representation of query words semantically, but they do not attempt to understand the user's query intent, but find words that are similar to each word for query expansion, which can easily lead to problems such as topic deviation and introduction of noise.

Images