SimBlock algorithm for realizing high-quality text similarity calculation and implementation method

Abstract

The method aims at big data text duplication removal scenes, such as search engine, information thematic module aggregation, identification of content originality and content plagiarism, content governance related to repeated irrigation posts and comments and the like. According to the technical scheme, the SimBlock algorithm (similar block matrix algorithm) capable of achieving similarity calculation is provided, the defect that ordered information is lost due to character intersection and character string vectorized cosine in a traditional similarity algorithm is overcome, local ordered information is supplemented, and the similarity calculation efficiency is improved. The given similarity score is not sensitive to the length of the two character strings, and the logic inclusion relation, the positions of the similar sub-character strings and the one-to-one correspondence relation can be judged. Another technical scheme of the invention is to provide a distributed computing technology architecture applicable to the algorithm, high-concurrency computing pressure is decomposed into a high-parallelism algorithm micro-service cluster, and high-concurrency read-write pressure is also decomposed into a cache cluster, so that a multi-process Source / Trans / Sink monomer is kept lightweight.

Description

technical field [0001] The present invention relates to a kind of SimBlock algorithm (similar block matrix algorithm) that realizes high-quality text similarity calculation, can give the similarity score between two character strings, logical containment relation, the position of similar substring and one by one Correspondence. The present invention also relates to a technical solution, applying the algorithm in the scene of deduplication of large data texts, and designing a distributed computing technical framework. Background technique [0002] Similarity algorithm is a basic service of search engine. The similarity algorithm aggregates search results with the same or similar text content (shallow character level or deeper semantic level) into a group and displays them as one search result, which not only saves content layout, improves search performance, but also saves users time and improve user experience. The similarity algorithm is widely used in scenarios such as ...

AI Technical Summary

Problems solved by technology

[0019] According to recent literature reports, Google's BERT model can effectively contain location information, design a multi-layer Multi-Head Attention deep learning network, and rely on dedicated hardware GPU / TPU for a large number of pre-training calculations, different network layers can represent different semantic levels , the semantic vectorization of different depths is its biggest advantage, but it is also impossible to mark similar substrings and give a one-to-one correspondence

Images