Efficient short text similarity determination method and device

Abstract

The invention relates to an efficient short text similarity determination method. The method comprises the following steps: performing word segmentation on short texts in a corpus to obtain corresponding word sequences; determining a penalty based on the total number of the short texts in the corpus, wherein the penalty is reduced along with the increase of the total number of the short texts in the corpus; determining a word frequency of each word in the word sequence and an adjusted inverse document frequency, wherein the adjusted inverse document frequency is calculated based on the penalty; weighting the word frequency of each word by using the adjusted inverse document frequency; combining the weighted word frequency of each word in the word sequence to determine a word frequency vector of the short text; and determining the similarity between the short text and other short texts based on the word frequency vector. The disclosure also relates to other related aspects.

Description

technical field [0001] The present application relates generally to natural language processing (NLP), and in particular to efficient short text similarity determination. Background technique [0002] Text similarity measurement is a common problem in the field of NLP. For long text and short text, academia and industry have also studied different measurement methods. [0003] For the similarity measurement method of long text, there are usually two paradigms: 1. Carry out vectorized representation of words or words, and calculate the similarity after aggregation to obtain the vector representation of long text. Common ones are: Word2vec, Bow model, etc.; 2. Introduce the deep learning network structure, according to the context semantics, learn the vector of the sentence or text, the common ones are Elmo, Bert, etc., and directly calculate the similarity through the constructed sentence vector. [0004] For short text similarity measurement methods, there are usually two ...

AI Technical Summary

Problems solved by technology

[0005] However, there are several problems in the similarity measurement for short texts, including: 1) the character-level similarity measurement ignores the word order factor; 2) for large-scale corpora, the vector dimension of one-hot increases linearly, and at the same time, data sparseness leads to similarity The distinction is not obvious, and for a small-scale corpus, the calculation formula of TF-idf will cause the idf value of low-frequency words to be too large, there are bumps in the text vector, and the similarity vibration is large

Images