A method and a device for calculating the weight of a word segmentation item

Abstract

The embodiment of the invention discloses a method and a device for calculating the weight of a segmented word item, which are used for accurately predicting the weight of each segmented word item ina query word. The embodiment of the invention provides a segmented word item weight calculation method, which comprises the following steps: carrying out word vector training by using text corpus andhistorical query words to obtain word vectors of the historical query words, and obtaining the historical query words through historical search data; taking The word vectors of the historical query words as features; on the basis of historical behavior data and a target value calculated by a recall result obtained by searching historical query words, training a plurality of segmented word item weights depending on the historical query words by using a machine learning algorithm, and ending the training when the error is minimum or the iteration frequency reaches a frequency threshold value; And using the machine learning algorithm to calculate the weights of the word segmentation items of the target query word, and outputting weight values of a plurality of word segmentation items depending on the target query word.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for calculating the weight of word segmentation items. Background technique [0002] In the search engine, the user can input a query word (query), and after performing word segmentation on the query word, multiple word segmentation terms (term) can be obtained. When a user enters a query word, the goal is to obtain useful information related to the query word. A good search engine is to accurately return the information the user is looking for and rank them first. The recall of the document is based on the intersection of each term in the query in the document. If the query is too long, some documents may not be recalled correctly and displayed to the user. Therefore, it is necessary to calculate the weight of each term in the query. According to the weight Wait for processing to recall and sort the documents. As an effective module, term weight is very i...

AI Technical Summary

Problems solved by technology

[0003] In the prior art, when calculating the weight of each term in the query word, the main calculation method is to obtain the relevant co-occurrence statistical features of the word from the multi-text data set, such as the common term frequency-inverse document frequency (Term Frequency-Inverse Document Frequency) frequency, TF-IDF), mutual information and other features, the relevant co-occurrence statistical features provided by the existing technology only consider the co-occurrence of words in the text and other information, and these information are independent of the query words themselves, making the The calculation result of term weight does not reflect the importance of the same term in different query words

Images