Text processing method and system and medium

Abstract

The invention discloses a text processing method and system and a medium. The method comprises the following steps: acquiring an automobile user comment text; performing word segmentation and stop word removal processing on the automobile user comment text; performing text keyword extraction on the text subjected to word segmentation and stop word removal to obtain a keyword extraction result; constructing a corresponding similarity vector space, and performing vectorization processing on the automobile user comment text to obtain an ultrahigh-dimensional high-dimensional vector; based on the high-dimensional vector, performing unbalanced cosine similarity analysis on the automobile user comment text to obtain phrase similarity; if the phrase similarity is greater than a preset threshold value, taking the automobile user comment text as a to-be-deleted text; otherwise, reserving the automobile user comment text. The text deduplication robustness can be improved, the defect that repeated contents in short texts and ultra-short texts such as automobile comments are extremely easy to be eliminated by mistake is overcome, and the problem that long sentences and short sentences which are repeated in meaning cannot be distinguished when a traditional cosine vector encounters is solved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a text processing method, system and medium. Background technique [0002] At present, the method of extracting keywords for car after-sales problems and forum user comments is mostly the MD5 method of content absolute mapping and the method of cosine similarity analysis of high-dimensional space vectors by local sensitive hashing. Among them, the traditional method uses MD5 to detect the same text content, The efficiency is very high, but a slight change in the characters will cause the identification of repeated keywords; the local sensitive hash method, regardless of the order of different sentence patterns, finds local close words and then determines the weight by the Hamming distance to check duplicates in long text segments However, it is very easy to mistakenly exclude such short texts and ultra-short texts such as car reviews, and the robustness of dedupli...

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to provide a text processing method, system and medium, which can improve the robustness of text deduplication, avoid the defects of short and ultra-short texts such as car reviews that easily cause repeated content to be excluded by mistake, and solve the traditional Cosine vector encounters the problem of indistinguishable long sentences and short sentences with repeated meanings

Images