Method and device for identifying coding mode of Chinese characters

Abstract

The embodiment of the invention provides a method and a device for identifying the coding mode of a Chinese character sequence. The method comprises the following steps: obtaining character sequence characteristics from the Chinese character sequence to be identified, wherein the character sequence characteristics comprise the length characteristic of the character sequence, or the length characteristic of the character sequence and the abnormal coding point characteristic of the character sequence; determining the coding mode of the Chinese character sequence to be identified based on a set Chinese coding identification strategy according to the character sequence characteristics. Complicated character probability distribution models do not need to be established in advance, and thus the identification process of identifying the coding mode is simplified; the detection range is shortened by adopting the length characteristic for the Chinese character sequence to be identified in mass network data, the occupation of internal storage caused by direct one-by-one detection on the abnormal coding points in the Chinese character sequence to be identified is avoided, the characteristics of abnormal coding points are further combined after the detection range is narrowed down, and the identification efficiency and accuracy rate of the coding mode are improved.

Description

technical field [0001] The embodiment of the present invention relates to the technical field of computer data communication, and in particular to a method and device for identifying a coding mode of Chinese characters. Background technique [0002] With the continuous development of computer communication technology, people have created a variety of encoding methods for transmitting data in the network. For Chinese characters, the commonly used encoding methods are GBK, GB2312 and UTF-8. After obtaining the encoded Chinese character sequence transmitted in the network, it is necessary to decode the obtained Chinese character sequence in order to correctly restore the original data corresponding to the Chinese character sequence. Therefore, the technology for identifying the encoding method of the Chinese character sequence Came into being. [0003] Existing recognition technologies for encoding modes of Chinese character sequences mainly include: encoding pattern method an...

AI Technical Summary

Problems solved by technology

[0004] The encoding mode method is mainly based on the encoding range to determine the encoding method. For a large number of character sequences, it can only be detected one by one based on the encoding range. Not only is the detection efficiency low, but also when there are a large number of encoding overlap points, it will be impossible to decide which encoding method to use.

[0005] The character distribution method is based on the character distribution probability as a model. Before identifying the encoding method, it is necessary to establish a character probability distribution model for a specific character set. For the intricate network environment, Chinese, English and other special symbols are often mixed together in the network data stream. When English characters and other non-Chinese characters are in the majority, it often interferes with the recognition of Chinese character encoding methods, especially affecting the recognition based on Recognition Accuracy of Character Encoding Recognition Scheme Based on Character Probability Distribution Model

Images