Coding format identification method, electronic equipment, storage medium and program product

Abstract

The invention provides a coding format recognition method, electronic equipment, a storage medium and a program product. The coding format recognition method comprises the steps of obtaining a to-be-detected coded text; according to the coding interval of the to-be-detected coded text, judging whether the coding interval belongs to the coding range of at least two coding formats or not; if yes, determining the weight sum of the to-be-detected coded text corresponding to each coding format according to the weight values corresponding to different codes in the coding weight library corresponding to each coding format; and determining a target coding format of the to-be-detected coded text according to the weight sum of the to-be-detected coded text corresponding to each coding format. The gbk coding format and the big5 coding format can be accurately distinguished, and the working efficiency and the coding format recognition accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method for identifying a coding format, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] With the rapid development of the Internet and the popularity of programming, data security has become an indispensable part of modern society. There are more and more cases of sending text at work, resulting in more and more confidential data leaks and losses to company property. Therefore, how to quickly and accurately identify the encoding format of the text can accurately know the true meaning of this Chinese character. Therefore, how to accurately identify the encoding format of Chinese characters is very important.

[0003] Currently, there are existing solutions for identifying text encoding, but they all have certain problems:

[0004] For example, Utf-8 encoding format (Universal Character Set / Unicode Transformation Format, a variable-length byte encoding method) is three-byte data, GBK encoding format (Chinese Internal Code Specification, full name "Chinese Character Internal Code Extension Specification"), Big5 encoding format (also known as Big Five or Big Five, a character encoding standard for Traditional Chinese) is double-byte data, and the byte ranges of the two encoding data are different, so it is possible to distinguish whether the Chinese characters in the text are in UTF-8 encoding format, GBK encoding format or Big5 encoding format.

[0005] However, since the byte ranges of the gbk encoding format or the big5 encoding format overlap, when the encoding of a text belongs to both the byte range of the gbk encoding format and the byte range of the big5 encoding format, the encoding format of the text cannot be accurately determined. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for identifying a coding format to solve the problem that the coding format of a text cannot be accurately identified in the prior art.

[0007] The above technical objectives of the present invention are achieved through the following technical solutions:

[0008] A method for identifying a coding format includes: obtaining a coded text to be detected; judging whether the coding interval belongs to the coding range of at least two coding formats according to the coding interval of the coded text to be detected; if so, determining the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format; determining the target coding format of the coded text to be detected according to the weight sum of the coded text to be detected corresponding to each coding format.

[0009] The embodiment of the present application sets a method for identifying a coding format. When determining whether a coding interval belongs to a coding range of at least two coding formats, the weight sum of the coded text to be detected corresponding to each coding format is calculated, and the target coding format is determined based on the weight sum. On the one hand, the target coding format of the text to be detected can be accurately identified. On the other hand, the program steps are simplified and work efficiency is improved.

[0010] Furthermore, the method also includes: if the coding interval does not belong to the coding range of at least two coding formats, determining the target coding format of the coded text to be detected according to the coding range to which the coding interval belongs and the coding format corresponding to the coding range.

[0011] In the embodiment of the present application, when the coding intervals do not all fall within the coding ranges of the two coding formats, the target coding format is determined directly by the coding range to which the coding intervals belong and the coding format corresponding to the coding ranges, thereby reducing the steps of obtaining the target coding format through complex calculations in the prior art, simplifying the procedure, improving work efficiency, and thereby improving the accuracy of coding format recognition.

[0012] Furthermore, according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format, the weight sum of the coded text to be detected corresponding to each coding format is determined, including: for each coding format, according to the corresponding weight value in the coding weight library corresponding to the coding format for each code in the coded text to be detected, the weight sum of all codes of the coded text to be detected is calculated.

[0013] Furthermore, according to the weight sum of the coded text to be detected corresponding to each coding format, the target coding format of the coded text to be detected is determined, including: according to the weight sum of the coded text to be detected corresponding to each coding format, the coding format corresponding to the maximum weight sum is taken as the target coding format of the coded text to be detected.

[0014] The embodiment of the present application obtains the weight sum by calculating the weight value of each code in the coding weight library corresponding to the coding format, and determines the target coding format by comparing the weight sum. Compared with the prior art, the calculation of the weight sum can more intuitively show the frequency of occurrence of the text to be detected in each coding format, and can also improve the accuracy of coding format judgment while simplifying the frequency calculation - that is, the calculation procedure of the weight value.

[0015] Furthermore, before determining the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format, the method also includes: obtaining multiple coded texts of each coding format; for each coding format, counting the word frequency of each word code in the multiple coded texts corresponding to the coding format; according to the word frequency of each word code corresponding to each coding format, obtaining the weight value of each word code corresponding to each coding format; according to the weight value of each word code corresponding to each coding format, constructing a coding weight library corresponding to the coding format.

[0016] The embodiment of the present application improves the calculation efficiency of weight values ​​by pre-building a coding weight library, that is, further improves the efficiency of coding format recognition.

[0017] Furthermore, before counting the frequency of each word in the multiple coded texts corresponding to each coding format, the method further includes: removing meaningless connecting words in the coded text according to a preset meaningless connecting word library.

[0018] The embodiment of the present application first removes meaningless conjunctions in the encoded text to further improve the calculation of weight values, and its encoding recognition accuracy is higher.

[0019] Furthermore, based on the coding interval of the coded text to be detected, it is determined whether the coding interval belongs to the coding range of at least two coding formats, including: based on the coding interval of the coded text to be detected, it is determined whether the coding interval belongs to the coding range of both the gbk coding format and the big5 coding format.

[0020] The embodiment of the present application distinguishes between the gbk encoding format and the big5 encoding format by setting an encoding format recognition method, which is more flexible than the prior art.

[0021] The present application also provides a coding format recognition device, including:

[0022] A text acquisition module is used to acquire the coded text to be detected;

[0023] A range determination module, used to determine whether the coding interval belongs to the coding range of at least two coding formats according to the coding interval of the coded text to be detected;

[0024] A weight determination module, used for determining the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format when the coding interval belongs to the coding range of at least two coding formats;

[0025] The format determination module is used to determine the target encoding format of the encoding text to be detected according to the weight sum of the encoding text to be detected corresponding to each encoding format.

[0026] An embodiment of the present application also provides an electronic device, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute any one of the above-mentioned coding format recognition methods.

[0027] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete any of the above-mentioned methods for identifying the encoding format.

[0028] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements any of the above-mentioned encoding format recognition methods when executed by a processor.

[0029] The beneficial effects of the embodiments of the present invention are:

[0030] The coding format recognition method provided in the embodiment of the present application determines whether the coding interval of the coded text to be detected is within the coding range of at least two coding formats. When both are within the coding range, the corresponding target coding format is determined by obtaining the weight sum according to the corresponding weight values ​​in the coding weight library preset for different coding formats, thereby accurately distinguishing the GBK coding format and the Big5 coding format, improving work efficiency and the accuracy of coding format recognition. Secondly, the coding weight library is pre-set and can be updated in real time. It can timely update the data that has not been saved in the coding weight library, and can be applied to the addition of new coding formats, which is flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0032] Figure 1 A schematic diagram of a flow chart of a method for identifying a coding format according to an embodiment of the present application;

[0033] Figure 2 This is a flow chart of a method for identifying a coding format according to another embodiment of the present application;

[0034] Figure 3A schematic diagram of a process of constructing a coding weight library for a coding format recognition method according to an embodiment of the present application;

[0035] Figure 4 A schematic diagram of a process of constructing a coding weight library for a coding format recognition method according to another embodiment of the present application;

[0036] Figure 5 It is a schematic diagram of a framework of a coding format recognition device provided in one embodiment of the present application;

[0037] Figure 6 It is a schematic diagram of a framework of a coding format recognition device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0038] In the description of this application, it should be noted that the terms "inside", "outside", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the application is usually placed when in use, which is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0039] The object of the present invention is to provide a method for identifying a coding format, which can solve the problem in the prior art that the coding of a text can only be distinguished as a three-byte coding format or a two-byte coding format, such as only being able to distinguish whether it is a UTF-8 coding format, a GBK coding format or a Big5 coding format, but cannot distinguish whether it is a GBK coding format or a Big5 coding format, and can only recognize simplified Chinese but cannot recognize traditional Chinese.

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0041] See also Figure 1 , the embodiment of the present application provides a method for identifying a coding format, including:

[0042] Step S110: Obtain the coded text to be detected.

[0043] The coded text to be detected is composed of the codes corresponding to each Chinese character.

[0044] Step S120: judging whether the coding interval is within the coding range of at least two coding formats according to the coding interval of the coded text to be detected.

[0045] The coding interval refers to the interval consisting of the maximum and minimum coding values ​​in the entire coded text to be detected. If the coding interval belongs to the coding range of any two coding formats at the same time, the coding format of the coded text to be detected cannot be directly determined.

[0046] In one embodiment, in order to distinguish whether the encoding format of the encoded text to be detected is the gbk encoding format or the big5 encoding format, the above step S120 may include: according to the encoding interval of the encoded text to be detected, determining whether the encoding interval belongs to the encoding range of the gbk encoding format and the big5 encoding format.

[0047] It is known that the encoding range of the gbk encoding format is 0x8140~0xFEFE, and the encoding range of the big5 encoding format is 0xA040~0xFE7E. It is determined whether the encoding interval of the text to be detected belongs to the above encoding range; of course, the two encoding formats are not limited to the above-mentioned gbk encoding format and big5 encoding format, and the above-mentioned encoding format recognition method can also be used to distinguish other encoding formats.

[0048] For example, if the code range of the code text to be detected includes 0xFEF1, which is within the code range of the gbk code format, then it can be directly determined that the code format of the code text to be detected is the gbk code format. If the code range of the code text to be detected is 0xA045-0xFE71, which is within the code range of 0x8140-0xFEFE and the code range of 0xA040-0xFE7E, then it is impossible to directly determine the code format of the code text to be detected.

[0049] Step S130: If yes, determine the sum of the weights of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format.

[0050] Among them, each encoding format has a corresponding encoding weight library. For example, the gbk encoding format has a corresponding encoding weight library, and the big5 encoding format has a corresponding encoding weight library. The encoding weight library corresponding to the gbk encoding format records the weight value of each encoding in the gbk encoding format. The encoding weight library corresponding to the big5 encoding format records the weight value of each encoding in the big5 encoding format. Among them, the encoding weight library can be constructed in advance, or it can be constructed in the manner described below, which will not be repeated here.

[0051] The weight sum may be the sum of the weight values ​​of each encoding in the encoded text to be detected.

[0052] In one embodiment, the above-mentioned step S130 specifically includes: for each encoding format, according to each encoding in the encoded text to be detected, the corresponding weight value in the encoding weight library corresponding to the encoding format, and calculating the weight sum of all encodings of the encoded text to be detected.

[0053] For example, assuming that the coded text to be detected is 0xb750 0xac56, 0xb750 and 0xac56 both belong to the encoding range of the gbk encoding format and the big5 encoding format, so it is impossible to determine whether the coded text to be detected is in the gbk encoding format or the big5 encoding format.

[0054] Therefore, based on the encoding weight library corresponding to the gbk encoding format, we can find the weight value of 0xb750 in the encoding weight library (assuming it is 0.05) and the weight value of 0xac56 in the encoding weight library (assuming it is 0.03). Then the sum of the weights of the encoded text to be detected corresponding to the gbk encoding format is 0.08.

[0055] Similarly, based on the encoding weight library corresponding to the big5 encoding format, the weight value of 0xb750 in the big5 encoding weight library (assuming it is 0.00005) and the weight value of 0xac56 in the big5 encoding weight library (assuming it is 0.00005) can be found. Then the sum of the weights of the encoded text to be detected corresponding to the big5 encoding format is 0.0001.

[0056] Step S140: Determine the target encoding format of the encoded text to be detected according to the weight sum of the encoded text to be detected corresponding to each encoding format.

[0057] The target encoding format refers to the accurate encoding format of the coded text to be detected obtained by identification. For the purpose of distinction, it is called the target encoding format.

[0058] In one embodiment, the above step S140 includes: according to the weight sum of the coded text to be detected corresponding to each coding format, taking the coding format corresponding to the maximum weight sum as the target coding format of the coded text to be detected.

[0059] For example, by comparing the weight sum corresponding to the gbk encoding format (e.g., 0.08) and the weight sum corresponding to the big5 encoding format (e.g., 0.0001) obtained in the above step S130, if 0.08 is greater than 0.0001, then it can be determined that the target encoding format of the coded text to be detected is the gbk encoding format. On the contrary, if the weight sum corresponding to the gbk encoding format is less than the weight sum corresponding to the big5 encoding format, then it is determined that the target encoding format of the coded text to be detected is the big5 encoding format.

[0060] The present application implements the recognition method for the above-mentioned encoding format. After determining that the encoding interval of the encoded text to be detected is within the encoding range of at least two preset encoding formats, the corresponding target encoding format is determined by obtaining the weight sum according to the corresponding weight value in the preset encoding weight library. Firstly, without the need for complicated calculation steps, the GBK encoding format and the Big5 encoding format can be accurately distinguished by comparing the size of the weight sum, thereby improving work efficiency and the accuracy of encoding format recognition. Secondly, the encoding weight library is pre-set and can be updated in real time. It can timely update the data that has not been saved in the library, and can be applied to texts with new encoding methods, which is flexible.

[0061] In one embodiment, if Figure 2 As shown, the encoding format recognition method also includes step S150: if the encoding interval does not belong to the encoding range of at least two encoding formats, determine the target encoding format of the encoded text to be detected according to the encoding range to which the encoding interval belongs and the encoding format corresponding to the encoding range.

[0062] Among them, in the above step S150, if the coding interval does not completely belong to the coding range of the big5 coding format and the coding range of the gbk coding format at the same time, then at this time, if the coding interval belongs to the coding range of the big5 coding format, it can be determined that the target coding format of the coded text to be detected is the big5 coding format, and if the coding interval belongs to the coding range of the gbk coding format, it can be determined that the target coding format of the coded text to be detected is the gbk coding format.

[0063] In the embodiment of the present application, when the coding ranges do not overlap, that is, the coding intervals do not belong to at least two coding ranges at the same time, the target coding format is determined directly by the coding range to which the coding interval belongs and the coding format corresponding to the coding range, thereby reducing the steps of obtaining the target coding format through complex calculations in the prior art, simplifying the procedure, improving work efficiency, and thereby improving the accuracy of coding format recognition.

[0064] The following is a detailed description of how to construct the encoding weight library, see Figure 3 Before determining the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format, that is, before the above step S130, the coding format recognition method further includes:

[0065] Step S510: Obtain multiple encoded texts in each encoding format.

[0066] For example, in one embodiment, Chinese text in the gbk encoding format and Chinese text in the big5 encoding format are obtained from news, books, technical materials, etc.

[0067] Step S520: For each encoding format, count the word frequency of each word encoding in the multiple encoded texts corresponding to the encoding format.

[0068] In one embodiment, for example, if a word encoding appears 3 times in the text in gbk encoding format, the word frequency of this word encoding in the gbk encoding format is counted as 3. If a word encoding appears 100 times in the text in big5 encoding format, the word frequency of this word encoding in the big5 encoding format is counted as 100.

[0069] Step S530: Obtain the weight value of each word encoding corresponding to each encoding format according to the word frequency of each word encoding corresponding to each encoding format.

[0070] In one embodiment, for example, a word encoding "0xb750" appears 3 times in the Chinese text in big5 encoding format, and there are 1000 word encodings in total in the Chinese text in big5 encoding format. Then the calculated weight value is 3 / 1000 = 0.003. Similarly, a word encoding "0xb750" appears 100 times in the Chinese text in gbk encoding format, and there are 500 words in total in the Chinese text in gbk encoding format. Then the calculated weight value is 100 / 500 = 0.2.

[0071] Step S540: Construct an encoding weight library corresponding to the encoding format according to the weight value of each word encoding corresponding to each encoding format.

[0072] In the above step S540, the weight value 0.003 corresponding to the above word encoding "0xb750" is stored in the encoding weight library corresponding to the big5 encoding format, and the weight value 0.2 corresponding to the word encoding "0xb750" is stored in the encoding weight library corresponding to the gbk encoding format.

[0073] Through the above step S540, the corresponding encoding weight library can be obtained, such as the encoding weight library corresponding to the gbk encoding format and the encoding weight library corresponding to the big5 encoding format.

[0074] Among them, in one embodiment, as Figure 4 shown, before the above step S520, the method further includes step S5110: Remove the meaningless connecting words in the encoded text according to the preset meaningless connecting word library, such as removing meaningless connecting words such as "de, ni, wo, ta".

[0075] Since the meaningless conjunctions appear too frequently in the text and have no practical meaning, they will not only increase the complexity of weight calculation, but also affect the accuracy of judging the text encoding format. Therefore, before counting the word frequency of each word encoding in the encoded text, this step is carried out first to remove the meaningless conjunctions, which can improve the accuracy of calculating the weights of other "keywords" and then improve the accuracy of encoding format judgment.

[0076] Specifically, for example, a word encoding appears 3 times in a Chinese text in the big5 encoding format. After removing the meaningless conjunctions from the Chinese text in the big5 encoding format, there are 300 words in total. Then the calculated weight value is 3 / 300 = 0.01. Similarly, a word encoding appears 100 times in a Chinese text in the gbk encoding format. After removing the meaningless conjunctions from the Chinese text in the gbk encoding format, there are 400 words in total. Then the calculated weight value is 100 / 400 = 0.25.

[0077] The weight values obtained in this way have high accuracy. At the same time, compared with the way of obtaining weight values through complex calculation processes in the prior art, the embodiments of the present application can improve work efficiency. Similarly, before calculating the sum of weights of the encoded text to be detected, the meaningless conjunctions can also be removed first, and then the sum of weights can be calculated further, thereby improving the accuracy of sum of weights calculation and then improving the accuracy of encoding format judgment.

[0078] The following will introduce the specific working process of the encoding format recognition method provided by an embodiment of the present application as follows:

[0079] In this process, taking "infection" as the encoded text to be detected and taking "de, ni, wo, ta" (meaningless conjunctions in Chinese) as an example for illustration, the specific content is determined according to the actual situation and is not specifically limited.

[0080] Collect a large number of Chinese texts in the gbk encoding format and the big5 encoding format from books, articles or technical materials.

[0081] For the gbk encoding format and the big5 encoding format respectively, read the collected Chinese texts, remove the meaningless conjunctions "de, ni, wo, ta" from them, obtain the remaining words, and calculate the weight value of each word encoding by calculating its word frequency. For example: after removing the meaningless conjunctions "de, ni, wo, ta", there are 100 remaining words, and the word "gan" appears 5 times, then its weight value is 5 / 100 = 0.05.

[0082] Calculate the word frequency of each word encoding in the above way, and then obtain the weight value, and construct an encoding weight library corresponding to the gbk encoding format and an encoding weight library corresponding to the big5 encoding format.

[0083] In this embodiment, by processing Chinese text in big5 encoding format, a coding weight library corresponding to big5 encoding format is constructed, and by processing Chinese text in gbk encoding format, a coding weight library corresponding to gbk encoding format is obtained.

[0084] Get the coded text to be detected. If the coded text to be detected is the word "infection", its hexadecimal code is 0xb7500xac56. It is known that the coding range of the gbk coding format is 0x8140~0xFEFE, and the coding range of the big5 coding format is 0xA040~0xFE7E. It can be seen that the coding of "infection" belongs to both the coding range of the big5 coding format and the coding range of the gbk coding format, that is, the coding interval of the coded text to be detected belongs to the coding range of the above two coding formats. At this time, it is impossible to distinguish whether the coded text to be detected is in the gbk coding format or the big5 coding format. It is necessary to calculate the sum of the weights of the words represented by "infection" in the coding weight library corresponding to the gbk coding format and in the coding weight library corresponding to the big5 coding format, and then make the next judgment.

[0085] For example: calculate the weight values ​​of the words represented by 0xb750 and 0xac56 in the encoding weight library corresponding to the gbk encoding format, add them up to get the first weight sum of 0.0001, calculate the weight values ​​of the words represented by 0xb750 and 0xac56 in the encoding weight library corresponding to the big5 encoding format, and add them up to get the second weight sum of 0.08.

[0086] Comparing the first weight sum with the second weight sum, it is found that the second weight sum is greater than the first weight sum. At this time, the encoding format corresponding to the maximum weight sum is used as the target encoding format of the encoded text to be detected, and the target encoding format is determined to be the big5 encoding format, that is, the encoding format of the encoded text to be detected is the big5 encoding format.

[0087] By setting the above-mentioned encoding format recognition method, the present application can accurately and efficiently determine the specific encoding format of the encoded text to be detected, and by constructing a coding weight library and combining weights and comparisons, it can determine whether the specific encoding format is the big5 encoding format or the gbk encoding format, thereby solving the problem that the prior art fails to distinguish between the big5 encoding format and the gbk encoding format. At the same time, the corresponding encoding weight library can be updated in real time to adapt to the new encoding format, which has flexibility, improves efficiency, and reduces the possibility that the encoded text to be detected does not exist in the coding weight library.

[0088] Figure 5 FIG. 1 is a schematic diagram of a framework of a coding format recognition device provided in an embodiment of the present application. Figure 5As shown, the device includes: a text acquisition module 510, a range determination module 520, a weight determination module 530 and a format determination module 540.

[0089] The text acquisition module 510 is used to acquire the coded text to be detected.

[0090] The range determination module 520 is used to determine whether the coding interval of the coded text to be detected belongs to the coding range of at least two coding formats.

[0091] The weight determination module 530 is used to determine the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format when the coding interval belongs to the coding range of at least two coding formats.

[0092] The format determination module 540 is used to determine the target encoding format of the encoding text to be detected according to the weight sum of the encoding text to be detected corresponding to each encoding format.

[0093] Specifically, the weight determination module 530 is specifically used to: for each encoding format, according to each encoding in the encoded text to be detected, and the corresponding weight value in the encoding weight library corresponding to the encoding format, calculate the weight sum of all encodings of the encoded text to be detected.

[0094] The format determination module 530 is specifically used to: according to the weight sum of the coded text to be detected corresponding to each coding format, take the coding format corresponding to the maximum weight sum as the target coding format of the coded text to be detected.

[0095] In one embodiment, the format determination module 530 is further used to: when the coding interval does not belong to the coding range of at least two coding formats, determine the target coding format of the coded text to be detected according to the coding range to which the coding interval belongs and the coding format corresponding to the coding range.

[0096] The embodiment of the present application sets up a coding format recognition device, which can obtain the weight sum of the coded text to be detected according to the weight values ​​corresponding to different codes in the coding weight library corresponding to the coding formats when the coding interval of the coded text to be detected coincides with the coding range of at least two coding formats, and select the coding format corresponding to the maximum weight sum as the target coding format of the coded text to be detected by comparing the weight sum. When there is no overlap, the target coding format of the coded text to be detected can be directly determined by the coding range to which the coding format belongs and the coding format corresponding to the coding range, thereby improving the efficiency and accuracy of coding format recognition and having a certain degree of flexibility.

[0097] Figure 6 It is a schematic diagram of a framework of a coding format recognition device provided in another embodiment of the present application.

[0098] In one embodiment, if Figure 6 As shown, the encoding format recognition device also includes: a sample acquisition module 610, a word frequency statistics module 620, a weight calculation module 630 and an encoding library storage module 640.

[0099] The sample acquisition module 610 is used to acquire multiple encoded texts in each encoding format.

[0100] The word frequency statistics module 620 is used to count the word frequency of each word encoding in multiple encoded texts corresponding to each encoding format.

[0101] The weight calculation module 630 is used to obtain the weight value of each word code corresponding to each encoding format according to the word frequency of each word code corresponding to each encoding format.

[0102] The coding library storage module 640 is used to construct a coding weight library corresponding to the coding format according to the weight value of each word coding corresponding to each coding format.

[0103] The coding format recognition device set in the embodiment of the application itself constructs a coding weight library by calculating the weight value of the corresponding coded text, thereby improving the calculation efficiency of the weight value, that is, further improving the efficiency of coding format recognition.

[0104] An embodiment of the present application also provides an electronic device, which includes a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the above-mentioned encoding format recognition method.

[0105] In one embodiment, the electronic device includes: one or more processors, and one or more memories storing processor-executable instructions, wherein the processor is configured to execute the coding format recognition method provided in the above embodiment of the present application.

[0106] The processor can be a gateway, or a smart terminal, or a device with other forms of processing units having data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device and control other components in the electronic device to perform desired functions.

[0107] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete any of the above-mentioned methods for identifying the encoding format.

[0108] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements the above-mentioned encoding format recognition method when executed by a processor.

[0109] The memory may include one or more of the above-mentioned computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer storage medium, and the processor may run the program instructions to implement the above-mentioned encoding format recognition method. Various applications and various data may also be stored in the computer-readable storage medium, such as the application using and / or generating various data, etc.

[0110] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "disposed" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0112] It should be noted that, in the absence of conflict, the features in the embodiments of this application may be combined with each other.

[0113] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying a coding format, characterized in that: include: Get the encoded text to be detected; According to the coding interval of the coded text to be detected, determining whether the coding interval belongs to the coding range of at least two coding formats; If yes, determine the sum of the weights of the to-be-detected coded texts corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format; The target encoding format of the coded text to be detected is determined according to the weight sum of the coded text to be detected corresponding to each encoding format.

2. The method for identifying a coding format according to claim 1, characterized in that: The method further comprises: If the coding interval does not belong to the coding range of at least two coding formats, the target coding format of the coded text to be detected is determined according to the coding range to which the coding interval belongs and the coding format corresponding to the coding range.

3. The method for identifying a coding format according to claim 1, characterized in that: The step of determining the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format comprises: For each encoding format, according to each encoding in the encoded text to be detected, the corresponding weight value in the encoding weight library corresponding to the encoding format is used to calculate the weight sum of all encodings of the encoded text to be detected.

4. The method for identifying a coding format according to claim 1, characterized in that: The step of determining the target encoding format of the encoded text to be detected according to the weight sum of the encoded text to be detected corresponding to each encoding format includes: According to the weight sum of the coded text to be detected corresponding to each coding format, the coding format corresponding to the maximum weight sum is used as the target coding format of the coded text to be detected.

5. The method for identifying a coding format according to claim 1, characterized in that: Before determining the weight sum of the coded text to be detected corresponding to each coding format according to the weight values ​​corresponding to different codes in the coding weight library corresponding to each coding format, the method further includes: Get multiple encoded texts for each encoding format; For each encoding format, counting the frequency of each word encoding in the plurality of encoded texts corresponding to the encoding format; According to the word frequency of each word encoding corresponding to each encoding format, the weight value of each word encoding corresponding to each encoding format is obtained; According to the weight value of each word encoding corresponding to each encoding format, a coding weight library corresponding to the encoding format is constructed.

6. The method for identifying a coding format according to claim 5, characterized in that: Before counting the word frequency of each word encoding in the plurality of encoded texts corresponding to the encoding format for each encoding format, the method further includes: According to a preset meaningless connection word library, meaningless connection words in the encoded text are removed.

7. The method for identifying a coding format according to claim 1, characterized in that: The step of judging whether the coding interval belongs to the coding range of at least two coding formats according to the coding interval of the coded text to be detected includes: According to the coding interval of the coded text to be detected, it is determined whether the coding interval belongs to the coding range of the gbk coding format and the big5 coding format.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method for identifying the coding format according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the method for identifying the encoding format described in any one of claims 1-7.

10. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the method for identifying the encoding format as described in any one of claims 1 to 7.

Images