Chinese character encoding method and device

Abstract

Embodiments of the present invention provide a method and device for identifying a coding mode of a Chinese character sequence. The method includes: obtaining character sequence features from the Chinese character sequence to be recognized, the character sequence features including: the length feature of the character sequence, or the length feature of the character sequence and the abnormal code point feature of the character sequence; according to the character sequence feature , based on the set Chinese encoding recognition strategy, determine the encoding method of the Chinese character sequence to be recognized. Since there is no need to establish a complex character probability distribution model in advance, the recognition process of the recognition encoding method is simplified; for the Chinese character sequences to be recognized in massive network data, the length feature is used to narrow the detection range, avoiding the need to directly recognize the Chinese character sequences one by one Detecting the memory occupation caused by abnormal code points, after narrowing the detection range, further combining the characteristics of abnormal code points, improves the recognition efficiency and accuracy of the coding method.

Description

technical field [0001] The embodiment of the present invention relates to the technical field of computer data communication, and in particular to a method and device for identifying a coding mode of Chinese characters. Background technique [0002] With the continuous development of computer communication technology, people have created a variety of encoding methods for transmitting data in the network. For Chinese characters, the commonly used encoding methods are GBK, GB2312 and UTF-8. After obtaining the encoded Chinese character sequence transmitted in the network, it is necessary to decode the obtained Chinese character sequence in order to correctly restore the original data corresponding to the Chinese character sequence. Therefore, the technology for identifying the encoding method of the Chinese character sequence Came into being. [0003] Existing recognition technologies for encoding modes of Chinese character sequences mainly include: encoding pattern method an...

AI Technical Summary

Problems solved by technology

[0004] The encoding mode method is mainly based on the encoding range to determine the encoding method. For a large number of character sequences, it can only be detected one by one based on the encoding range. Not only is the detection efficiency low, but also when there are a large number of encoding overlap points, it will be impossible to decide which encoding method to use.

[0005] The character distribution method is based on the character distribution probability as a model. Before identifying the encoding method, it is necessary to establish a character probability distribution model for a specific character set. For the intricate network environment, Chinese, English and other special symbols are often mixed together in the network data stream. When English characters and other non-Chinese characters are in the majority, it often interferes with the recognition of Chinese character encoding methods, especially affecting the recognition based on Recognition Accuracy of Character Encoding Recognition Scheme Based on Character Probability Distribution Model

Images