Text similarity calculation deduplication method and system, medium and equipment

Abstract

The invention discloses a text similarity calculation deduplication method and system, a medium and equipment, and the method comprises the steps: importing a hotline text data set into a database, inputting the hotline text data set into a trained model, carrying out the feature extraction, feature fusion and feature enhancement, obtaining the final features of the data, and carrying out the serialization storage; inputting to-be-calculated data into the model to extract final features; carrying out cosine similarity calculation on the hot line text data and the hot line text data in the corresponding date to obtain a similarity result; and outputting a similarity result of the ranking top50, and performing duplicate removal. The method is oriented to the government affair service convenience hotline, the final sentence features of the hotline text content are extracted through the model for similarity calculation and deduplication, an external calling way is further provided, different model functions can be achieved through four interface operation models, operation is convenient, practicability is high, and the method is suitable for the hotline field.

Description

technical field [0001] The invention belongs to the technical field of text similarity, and in particular relates to a method, system, medium and device for calculating and deduplicating text similarity. Background technique [0002] In the era of intelligence, natural language processing technology has developed rapidly and has been widely used. In the application field, it often involves how to measure and calculate the similarity between two texts. Text similarity is widely used in many Domains, such as knowledge retrieval, text duplication detection, automatic generation of text summaries, recommender systems, text classification, machine translation, etc. Text similarity is a very important and indispensable technique in natural language processing. [0003] Text similarity refers to the similarity between two texts. The text is in a high-dimensional semantic space, and it needs to be abstractly decomposed to be able to quantify the similarity between them from a mathe...

AI Technical Summary

Problems solved by technology

[0005] In recent years, text similarity calculation methods have been continuously proposed, but most of these methods are widely used in various fields, with insufficient accuracy and lack of pertinence. For the hotline field, these methods will introduce too much noise and the effect Not very ideal, and, in the hotline field where text similarity plays a very important role, there is currently no ideal text similarity calculation method applied, and a good deduplication effect cannot be achieved

Images