Chinese word similarity detection algorithm based on pronunciation, shape and meaning

Abstract

The invention provides a Chinese word similarity detection algorithm based on pronunciation, shape and meaning, which detects the overall similarity of Chinese character strings by comprehensively considering three characteristics of pronunciation, shape and meaning of Chinese characters, and comprises the following steps of: firstly, converting the pinyin of each Chinese character of the Chinesecharacter strings s1 and s2 into a corresponding phonetic code, and converting each Chinese character of the Chinese character strings s1 and s2 into a shape code; then respectively calculating the phonetic code similarity and the shape code similarity between the Chinese character strings s1 and s2, then independently calculating the similarity of the Chinese character string meanings, and finally setting contribution parameters for an application scene in combination with the phonetic form meanings to calculate the overall similarity of the final Chinese character strings s1 and s2. The algorithm can meet complex application scenarios, can be applied to detection of the repetition degree of structured data items, especially in the case of manual input errors, and can also be applied to detection of sensitive words hidden in wrongly written characters and the like. Compared with a Chinese character similarity detection algorithm of the same type, the detection effect on the Chinese character string similarity is greatly enhanced.

Description

technical field [0001] The invention relates to the technical field of Chinese word similarity, and more specifically, to a Chinese word similarity detection algorithm based on sound, form and meaning. Background technique [0002] The string similarity algorithm refers to a certain method to calculate the similarity between two different strings. A percentage is usually used to measure the similarity between strings. String similarity algorithms are used in many computing scenarios, such as data cleaning, user input error correction, recommendation systems, plagiarism detection systems, automatic scoring systems, as well as web search and DNA sequence matching. At present, the commonly used algorithms for similarity detection of Chinese character strings are as follows: one is based on the similarity detection of Chinese characters, by obtaining the basic information of Chinese characters, such as the pinyin of Chinese characters, the shape structure, the number of strokes...

AI Technical Summary

Problems solved by technology

At present, the commonly used algorithms for the similarity detection of Chinese character strings are: first, the similarity detection based on the phonetic shape of Chinese characters, by obtaining the basic information of Chinese characters, such as the pinyin of Chinese characters, the shape structure, the number of strokes, the order of strokes, etc. The data generates mathematical expressions according to certain coding rules, and then uses specific algorithms to process the mathematical expressions to obtain the similarity of Chinese characters; the second is the similarity detection based on the semantics of Chinese characters, by comparing Chinese character strings with those included in a large knowledge base Words and descriptions are compared, and then the semantic similarity of Chinese characters is calculated according to the distance of the sememe in the knowledge base; however, these two types of methods have defects. The detection method must be based on the fact that the detected words are completely correct, and the similarity between words with hidden typos cannot be detected

Images