Text similarity calculation method and apparatus

Abstract

The invention provides a text similarity calculation method. The method comprises the steps of performing text segmented word filtering processing on text segmented words obtained by performing word segmentation processing on a text sample in an original black sample library and a newly input text sample according to a text filtering ratio of multiple preserving gradients based on a same drop policy; performing reconstruction on the text sample in the original black sample library and the newly input text sample by using the residual text segmented words after the filtering; representing the similarity between the newly input text sample and the black sample by utilizing the filtering ratio of the text segmented words; and by matching the text segmented words in the reconstructed black sample library and newly input text sample, setting black sample similarity for the text segmented words obtained by performing word segmentation on the newly input text sample. According to the method, the calculation efficiency of calculating the similarity between the newly input text sample and the text sample in the black sample library can be remarkably improved.

Description

technical field [0001] The present application relates to the field of computer applications, in particular to a method and device for calculating text similarity. Background technique [0002] Social applications usually face the problem of content review. A social product usually has tens of millions or even hundreds of millions of users, and there is a huge amount of information interacting every day and every moment. Therefore, how to quickly complete the real-time online prevention and control of various harmful content based on the audited historical content is of great significance. [0003] In related technologies, real-time online prevention and control of various bad content based on reviewed bad historical content is usually based on text similarity; for example, based on algorithms such as edit distance or cosine distance, Calculate the text similarity between the text samples generated by social applications and every black sample containing bad content that h...

AI Technical Summary

Problems solved by technology

[0004]However, through algorithms such as edit distance or cosine distance, when calculating the similarity between the text sample generated by social text and each black sample, it usually faces 1: N polling; therefore, when the number of black samples is large, all black samples are polled to calculate the similarity in turn. From the perspective of response speed, it cannot meet the requirements of real-time online prevention and control.

Images