Method for processing rare character garbled code in data file and related device

Abstract

The application discloses a processing method and related device for rare character garbled code in a data file, and relates to the field of data recovery, which comprises: character coding of each Chinese character, each character coding of Chinese character is processed by query, the query processing comprises: according to the arrangement order of the bytes included in the character coding, each byte is sequentially taken as a current byte, the coding range in which the byte group composed of the current byte and the byte before the current byte falls is determined, if it falls into the garbled code range, the Chinese character is determined as garbled code, and each byte before the current byte is extracted; the continuously arranged garbled code is determined as a garbled code group, the extracted each byte of each garbled code group and the high-frequency rare character words included in the preset high-frequency rare character word library are processed by using an association rule mining algorithm, the rare character word corresponding to the garbled code group is obtained, and the garbled code group is replaced by the rare character word corresponding to the garbled code group. The application avoids the influence of the data file use by these garbled codes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data recovery, in particular to a processing method for rare character garbled code in a data file and related devices. BACKGROUND

[0002] With the popularization of office electronization, the use of data files such as forms is becoming more and more popular, and the types and contents of data in data files are also increasing.

[0003] In a data file, there may be some rare characters, which are prone to become garbled codes during storage and transmission, resulting in the use of data files being affected by these garbled codes. SUMMARY

[0004] In view of the above problems, the present application provides a processing method for rare character garbled code in a data file and related devices to restore the garbled code to the rare character. The specific scheme is as follows:

[0005] The first aspect of the present application provides a processing method for rare character garbled code in a data file, comprising:

[0006] Positioning the positions of each Chinese character included in the data file, extracting the character code of each Chinese character based on the position, and performing query processing on the character code of each Chinese character, the query processing comprising: according to the arrangement order of the bytes included in the character code, sequentially taking each byte as the current byte, determining the coding range in which the byte group composed of the current byte and the byte before the current byte falls, if the coding range falls within the garbled code range, determining that the Chinese character is garbled, and extracting each byte before the current byte and stopping sequentially taking each byte as the current byte.

[0007] Determine the continuously arranged garbled code as a garbled code group, use the association rule mining algorithm to process each byte extracted from each garbled code group and the high-frequency rare character words included in the preset high-frequency rare character word library, obtain the rare character words corresponding to the garbled code group, and replace the garbled code group with the rare character words corresponding to the garbled code group based on the position of the garbled code group.

[0008] In one possible implementation, the processing of each byte extracted from each garbled code group and the high-frequency rare character words included in the preset high-frequency rare character word library comprises:

[0009] Processing the bytes falling within the rare character range in each byte extracted from each garbled code group and the high-frequency rare character words included in the preset high-frequency rare character word library.

[0010] In a possible implementation, the association rule mining algorithm is an Apriori algorithm, and the association rule mining algorithm is used to process each extracted byte of the messy code word group and a high-frequency rare character word library to obtain a rare character word corresponding to the messy code word group, including:

[0011] The Apriori algorithm is used to generate a plurality of optional rare character combinations corresponding to each extracted byte of the messy code word group, each rare character in the optional rare character combination being a rare character in the preset high-frequency rare character word library;

[0012] The Apriori algorithm is used to determine the support of each optional rare character combination, and the optional rare character combination with a support lower than a threshold is discarded;

[0013] The Apriori algorithm is used to generate an association rule based on the remaining optional rare character combinations;

[0014] The Apriori algorithm is used to determine the confidence of each remaining optional rare character combination based on the association rule;

[0015] The Apriori algorithm is used to determine the optional rare character combination with the highest confidence as the rare character word corresponding to the messy code word group.

[0016] In a possible implementation, after the messy code word group is replaced by the rare character word corresponding to the messy code word group, the position of each Chinese character included in the positioning data file is returned to be executed, and the character code of each Chinese character is extracted based on the position until no Chinese character is determined to be messy code.

[0017] In a possible implementation, the messy code arranged continuously is determined as a messy code word group, including:

[0018] The messy code arranged continuously is determined as a messy code word group.

[0019] If there is messy code not arranged continuously with other messy code, the messy code not arranged continuously with other messy code is determined as a messy code word group.

[0020] The second aspect of the present application provides a rare character messy code processing device in a data file, including:

[0021] The rare character garbled code checking module is configured to locate positions of each Chinese character included in the data file, extract character codes of each Chinese character based on the positions, and perform query processing on the character codes of each Chinese character, which includes: sequentially taking each byte as a current byte according to an arrangement order of the bytes included in the character codes, determining a code range in which a byte group composed of the current byte and the bytes before the current byte falls, and determining that the Chinese character is garbled code if the determined code range is a garbled code range, and extracting the bytes before the current byte and stopping the sequential taking of each byte as the current byte.

[0022] The rare character garbled code recovery module is configured to determine the continuously arranged garbled codes as a garbled code group, process each byte extracted from each garbled code group and high-frequency rare character words included in a preset high-frequency rare character word library by using an association rule mining algorithm, obtain a rare character word corresponding to the garbled code group, and replace the garbled code group with the rare character word corresponding to the garbled code group based on the position of the garbled code group.

[0023] In a possible implementation, the rare character garbled code recovery module processes each byte extracted from each garbled code group and high-frequency rare character words included in a preset high-frequency rare character word library, and is specifically configured to:

[0024] Process the bytes included in the rare character range in each byte extracted from each garbled code group and the high-frequency rare character words included in the preset high-frequency rare character word library.

[0025] The third aspect of the present application provides a computer program product, which includes computer readable instructions, when the computer readable instructions run on an electronic device, make the electronic device implement the processing method of the rare character garbled code in the data file of the first aspect or any implementation manner of the first aspect.

[0026] The fourth aspect of the present application provides an electronic device, which includes at least one processor and a memory connected with the processor, wherein:

[0027] The memory is configured to store a computer program;

[0028] The processor is configured to execute the computer program, so that the electronic device can implement the processing method of the rare character garbled code in the data file of the first aspect or any implementation manner of the first aspect.

[0029] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs, when the one or more computer programs are executed by an electronic device, can make the electronic device implement the processing method of the rare character garbled code in the data file of the first aspect or any implementation manner of the first aspect.

[0030] By the technical scheme, the method and the related device for processing the rare Chinese character code in the data file can locate positions of each Chinese character included in the data file, extract character codes of each Chinese character based on the positions, and perform query processing on the character codes of each Chinese character, the query processing including: according to an arrangement order of bytes included in the character codes, sequentially taking each byte as a current byte, determining a code range in which a byte group composed of the current byte and the bytes before the current byte falls, if the code range falls in a code range of a code, determining that the Chinese character is the code, and extracting the bytes before the current byte and stopping the sequential taking of each byte as the current byte; determining the continuously arranged codes as a code group, processing each byte extracted from each code group and high-frequency rare Chinese character words included in a preset high-frequency rare Chinese character word library by using an association rule mining algorithm, obtaining a rare Chinese character word corresponding to the code group, and replacing the code group with the rare Chinese character word corresponding to the code group based on the position of the code group. The application can identify the code based on the character code, extract the byte group not falling in the code range in the code of the code, obtain the rare Chinese character word corresponding to the code group by using the association rule mining algorithm and the preset high-frequency rare Chinese character word library, and then replace the code group with the rare Chinese character word corresponding to the code group, so that the application can restore the code to the rare Chinese character, and avoid the influence of the code on the use of the data file. BRIEF DESCRIPTION OF DRAWINGS

[0031] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail the following specific embodiments thereof with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic and elements and features are not necessarily to scale.

[0032] Figure 1 A system architecture schematic diagram is provided for the application;

[0033] Figure 2 An optional hardware structure schematic diagram of a terminal is provided for the application;

[0034] Figure 3 A structure schematic diagram of a server is provided for the application;

[0035] Figure 4 A flowchart of a method for processing rare Chinese character codes in a data file is provided for the embodiments of the application;

[0036] Figure 5 A common Chinese character code schematic diagram is provided;

[0037] Figure 6 A rare Chinese character code schematic diagram is provided;

[0038] Figure 7 A schematic diagram of a device for processing garbled characters in a data file, provided in an embodiment of this application;

[0039] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0041] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0042] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0043] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (The example includes a server), and the server 200 can provide the method provided in the embodiments of this application to one or more terminals.

[0044] The terminal 100 may have an application with data file recovery function installed. The application and webpage can provide an interface. The terminal 100 can obtain the data file selected by the user on the data file upload interface and send the data file to the server 200. The server 200 can obtain the processing result (the data file after replacing the garbled characters with uncommon characters) based on the received data file and return the processing result to the terminal 100.

[0045] It should be understood that, in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the obtained data file by itself without the cooperation of the server, and the embodiments of the present application are not limited thereto.

[0046] Next, the product form of the terminal 100 is described. Figure 1

[0047] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not make any limitation thereto.

[0048] Figure 2 An optional hardware structure schematic diagram of the terminal 100 is shown.

[0049] Referring to FIG. 1, Figure 2 As shown in FIG. 1, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a headset jack 163 (optional), a processor 170, an external interface 180, a power supply 190, etc. Those skilled in the art can understand that the terminal 100 can include more or less components, or combine certain components, or different components, and the embodiments of the present application are not limited thereto. Figure 2 The terminal or the multifunctional device is only an example and does not constitute a limitation on the terminal or the multifunctional device, and can include more or less components, or combine certain components, or different components.

[0050] ​The input unit 130 can be used to receive inputted digital or character information, and to generate key signal inputs related to user settings of the portable multifunctional device and control of functions. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of a user thereon or therearound (such as operations of the user using a finger, a knuckle, a stylus, or any suitable object on or near the touch screen), and drive corresponding connected devices according to pre-set programs. The touch screen can detect touch actions of the user on the touch screen, convert the touch actions into touch signals and send the touch signals to the processor 170, and can receive commands from the processor 170 and execute the commands; the touch signals at least include touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc.

[0051] The input device 132 can receive inputted data, etc.

[0052] The display unit 140 can be used to display information inputted by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playing of any kind of multimedia files. In the embodiments of the present application, the display unit 140 can be used to display a data file uploading interface, processing results, etc.

[0053] The storage 120 can be used to store instructions and data. The storage 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc.; the storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc., or their subsets, expanded sets. It can also include a non-volatile random access memory; provide the processor 170 with software and applications that include management of hardware, software, and data resources in the computing processing device, support control. It is also used for storage of multimedia files, and storage of running programs and applications.

[0054] The processor 170 is the control center of the terminal 100, connects each part of the whole terminal 100 by various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing the instructions stored in the memory 120 and calling the data stored in the memory 120, thereby performing overall control on the terminal device. Optionally, the processor 170 can include one or more processing units; preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 170 can also be used to generate corresponding operation control signals to send to corresponding components of the computing processing device, read and process data in the software, especially read and process the data and programs in the memory 120, so that each functional module therein executes corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0055] The memory 120 can be used to store software codes related to the processing method of the rare character garbled code in the data file, and the processor 170 can execute the steps of the processing method of the rare character garbled code in the data file, or can also dispatch other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0056] The RF unit 110 (optional) can be used to receive and send signals in the process of information or communication, for example, receiving the downlink information of the base station, and processing by the processor 170. In addition, the uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0057] In the embodiments of the present application, the RF unit 110 can send data to the server 200 and receive the processing result sent by the server 200.

[0058] It should be understood that the RF unit 110 is optional, which can be replaced by other communication interfaces, for example, a network interface.

[0059] The terminal 100 further includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize the functions of power management, such as charge management, discharge management and power consumption management, through the power management system.

[0060] The terminal 100 further includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector. The external interface 180 can be used to connect the terminal 100 with other devices for communication, and can also be used to connect a charger for charging the terminal 100.

[0061] Although not shown, the terminal 100 can further include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described here. Some or all of the methods described below can be applied to the terminal 100 as shown. Figure 2

[0062] ​Next, the product form of the server 200 is described. Figure 1 The product form of the server 200 is described.

[0063] Figure 3 A structural diagram of the server 200 is provided, as shown in the figure. Figure 3 The server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0064] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus.

[0065] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0066] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0067] The memory 204 can be used to store software code related to the processing method of the rare Chinese characters in the data file, and the processor 202 can execute the steps of the processing method of the rare Chinese characters in the data file of the chip, or can schedule other units to realize the corresponding functions.

[0068] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (for example, the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (for example, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processing (DSP), a microprocessor, a microcontroller, or the like) or a combination of the hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0069] To solve the above problems, the embodiment of the present application provides a processing method of rare character garbled code in a data file. The processing method of rare character garbled code in a data file of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0070] Referring to Figure 4 , Figure 4 The flow of the processing method of rare character garbled code in a data file provided by the embodiment of the present application is shown in FIG. 1. The processing method of data provided by the embodiment of the present application can include steps 401 to 402, which will be described in detail below. Figure 4

[0071] 401, locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character. The query processing includes: according to the arrangement order of the bytes included in the character code, sequentially taking each byte as a current byte, determining the code range in which the byte group composed of the current byte and the byte before the current byte falls, if the falling code range is a garbled code range, determining that the Chinese character is garbled, and extracting each byte before the current byte and stopping sequentially taking each byte as the current byte.

[0072] The data file is a file used to store data, and the type of the data file can be various, such as a table, a database file, and the like.

[0073] ​For some data files (such as the information table of a financial institution's business outlets), the format of the data carried by the data files is relatively fixed, and the positions of the Chinese characters in the data files are pre-set. For example, the positions, contents and formats of the fields of the information table of a financial institution's business outlets are fixed, the fourth column is the name of the legal person of the business outlet, and the name of the legal person is a Chinese character. Therefore, the application can determine that the data in each cell of the fourth column is a Chinese character.

[0074] In another optional embodiment, the application can also obtain the position information of the Chinese characters from the data file or outside the data file, for example, the position information of the Chinese characters is carried in the data file, or the position information of the Chinese characters in the data file is stored in another file.

[0075] Character encoding (Character encoding) is also called character set code, which encodes characters in a character set into an object in the character set (Character Set) so as to store and transmit text in a computer through a communication network. The character set is a mapping table of one-to-one correspondence between characters and character encodings, and common character sets include ASCII, Unicode, etc.

[0076] After determining the positions of the Chinese characters, the character encodings of the Chinese characters can be extracted from the corresponding positions. It should be noted that when the character encodings are extracted, the query processing has not been performed, and therefore it is not known whether the Chinese characters at the positions are displayed normally. If the Chinese characters are displayed normally, the extracted character encodings will be the encodings of the Chinese characters. If the Chinese characters are displayed as garbled codes, the extracted character encodings will be the encodings of the garbled codes. In order to determine whether the Chinese characters are garbled, the application can perform query processing on the character encodings of each Chinese character respectively.

[0077] Taking Unicode as an example, as shown in Figure 5 , the Unicode encoding range of commonly used Chinese characters is \u4e00 to \u9FFF. As shown in Figure 6 , the Unicode encoding range of uncommon Chinese characters (i.e. rare characters) is \u20000 to \u2A6DF. The first letter u in the above encoding range represents Unicode encoding, and the bytes included in the character encoding using Unicode encoding are the contents after the letter u, which are generally four bytes or five bytes. For example, the character encoding of the above-mentioned Unicode encoding range of commonly used Chinese characters includes four bytes, and the encoding range of rare characters includes five bytes.

[0078] For convenience of understanding, the extracted character encoding u2A6FF is taken as an example for description:

[0079] First, according to the arrangement order of the five bytes 2, A, 6, F, F included in u2A6FF, 2 is determined as the current byte, and the code range into which the byte group composed of 2 and the byte before 2 falls is determined. Since In the Unicode code range of Chinese, the next Unicode code A is determined as the current byte, and the code range into which the byte group composed of A and the byte before A falls is determined. Since Still in the Chinese range, the next Unicode code 6 is determined as the current byte, and the code range into which the byte group composed of 6 and the byte before 6 falls is determined. In this way, the cycle is executed until the code range into which the byte group falls is not in the Chinese range or there is no corresponding code. In this example, since It is out of the Chinese range, that is, it is determined that the Chinese character is garbled, and each byte before the current byte is extracted, that is, .

[0080] 402. Determine the continuously arranged garbled codes as a garbled code group, and use the association rule mining algorithm to process each byte extracted from the garbled code group and the high-frequency rare Chinese characters included in the preset high-frequency rare Chinese character library to obtain the rare Chinese characters corresponding to the garbled code group. Based on the position of the garbled code group, the garbled code group is replaced by the rare Chinese characters corresponding to the garbled code group.

[0081] Generally, after the garbled code appears, the garbled codes are continuously arranged, so the application can determine the continuously arranged garbled codes as a garbled code group. Since the positions of the multiple continuously arranged garbled codes should be continuously arranged Chinese characters, and the continuously arranged Chinese characters have semantic continuity, for example, a Chinese word or even a Chinese short sentence. Therefore, if the garbled code group is analyzed, not only can the corresponding rare Chinese characters be found according to the character encoding of the garbled code itself, but also the corresponding rare Chinese characters can be found according to the relationship between the character encodings of the multiple garbled codes.

[0082] Optionally, the step 402 of determining the continuously arranged garbled codes as a garbled code group can include:

[0083] The continuously arranged garbled codes are determined as a garbled code group, and if there is a garbled code that is not continuously arranged with other garbled codes, the garbled code that is not continuously arranged with other garbled codes is determined as a garbled code group.

[0084] If there is a garbled code that is not continuously arranged with other garbled codes, the garbled code that is not continuously arranged with other garbled codes is determined as a garbled code group, and the corresponding rare Chinese characters are tried to be found.

[0085] In practical applications, a preset high-frequency rare Chinese character word library can be established according to actual businesses or industries. For example, for the financial industry, high-frequency rare Chinese characters commonly used in the financial industry can be combined into a preset high-frequency rare Chinese character word library. Of course, in addition to high-frequency rare Chinese characters, the preset high-frequency rare Chinese character word library can also include words composed of multiple high-frequency rare Chinese characters. In this application, the words composed of multiple rare Chinese characters are referred to as rare Chinese character words.

[0086] Specifically, the processing of each byte of the extracted rare Chinese character word group and the high-frequency rare Chinese character words included in the preset high-frequency rare Chinese character word library in step 402 can include:

[0087] Processing the bytes falling within the rare Chinese character range in each byte of the extracted rare Chinese character word group and the high-frequency rare Chinese character words included in the preset high-frequency rare Chinese character word library.

[0088] If it does not fall within the rare Chinese character range, it is possible that the Chinese character is not a rare Chinese character, and no processing is required.

[0089] Optionally, the association rule mining algorithm is the Apriori algorithm. The Apriori algorithm can use an iterative method of layer-by-layer search to find the relationship of item sets in the database to form rules. The process consists of connection (matrix operation) and pruning (removing unnecessary intermediate results).

[0090] Step 402 uses an association rule mining algorithm to process each byte of the extracted rare Chinese character word group and the high-frequency rare Chinese character words included in the preset high-frequency rare Chinese character word library, and obtains the rare Chinese character word corresponding to the rare Chinese character word group. It can specifically include:

[0091] Using the Apriori algorithm to generate multiple optional rare Chinese character combinations corresponding to each byte of the extracted rare Chinese character word group. Each rare Chinese character in the optional rare Chinese character combination is a rare Chinese character in the preset high-frequency rare Chinese character word library;

[0092] Using the Apriori algorithm to determine the support of each optional rare Chinese character combination, and discarding the optional rare Chinese character combination with a support lower than the threshold;

[0093] Using the Apriori algorithm to generate association rules based on the remaining optional rare Chinese character combinations;

[0094] Using the Apriori algorithm to determine the confidence of each remaining optional rare Chinese character combination based on the association rules;

[0095] Using the Apriori algorithm to determine the optional rare Chinese character combination with the highest confidence as the rare Chinese character word corresponding to the rare Chinese character word group.

[0096] Optionally, in another optional embodiment of the present application, after replacing the garbled character group with the rare Chinese character word corresponding to the garbled character group, the step of locating the position of each Chinese character included in the data file, extracting the character code of each Chinese character based on the position, is executed until no Chinese character is determined as garbled.

[0097] In this way, if the method shown in Figure 1 cannot recover all the garbled characters, or introduces new garbled characters, the method shown in Figure 1 can be continuously executed until there is no garbled character in the data file.

[0098] The above describes a method for processing garbled rare Chinese characters in a data file according to an embodiment of the present application. The following describes a device for executing the above method for processing garbled rare Chinese characters in a data file.

[0099] Please refer to Figure 7 , Figure 7 for a structural schematic diagram of a device for processing garbled rare Chinese characters in a data file according to an embodiment of the present application. As shown in Figure 7 , the device for processing garbled rare Chinese characters in a data file includes:

[0100] The garbled rare Chinese character checking module 701 is configured to locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character. The query processing includes: according to the arrangement order of the bytes included in the character code, sequentially taking each byte as a current byte, determining the code range in which the byte group formed by the current byte and the bytes before the current byte falls, determining the Chinese character as garbled if the falling code range is a garbled code range, and extracting the bytes before the current byte and stopping the sequential taking of each byte as the current byte.

[0101] The garbled rare Chinese character recovery module 702 is configured to determine the continuously arranged garbled characters as a garbled character group, process the extracted bytes of each garbled character group and the high-frequency rare Chinese character words included in the preset high-frequency rare Chinese character word library using an association rule mining algorithm, obtain the rare Chinese character word corresponding to the garbled character group, and replace the garbled character group with the rare Chinese character word corresponding to the garbled character group based on the position of the garbled character group.

[0102] Optionally, the garbled rare Chinese character recovery module processes the extracted bytes of each garbled character group and the high-frequency rare Chinese character words included in the preset high-frequency rare Chinese character word library, and is specifically configured to:

[0103] Process the bytes falling in the rare Chinese character range in the extracted bytes of each garbled character group and the high-frequency rare Chinese character words included in the preset high-frequency rare Chinese character word library.

[0104] Optionally, the association rule mining algorithm is an Apriori algorithm, and the rare character and code recovery module uses the association rule mining algorithm to process each byte extracted from the code word group and the high-frequency rare character words included in the preset high-frequency rare character word library, to obtain the rare character word corresponding to the code word group. The specific setting is as follows:

[0105] The Apriori algorithm is used to generate a plurality of optional rare character combinations corresponding to each byte extracted from the code word group. Each rare character in the optional rare character combination is a rare character in the preset high-frequency rare character word library. The Apriori algorithm is used to determine the support degree of each optional rare character combination, and the optional rare character combination with a support degree lower than a threshold value is discarded. The Apriori algorithm is used to generate an association rule based on the remaining optional rare character combinations. The Apriori algorithm is used to determine the confidence degree of each remaining optional rare character combination based on the association rule. The Apriori algorithm is used to determine the optional rare character combination with the highest confidence degree as the rare character word corresponding to the code word group.

[0106] Optionally, after the rare character and code recovery module replaces the code word group with the rare character word corresponding to the code word group, the rare character and code checking module is triggered to process until no Chinese character is determined as code.

[0107] Optionally, the rare character and code recovery module determines the continuously arranged code as one code word group. The specific setting is as follows:

[0108] The continuously arranged code is determined as one code word group. If there is code not continuously arranged with other codes, the code not continuously arranged with other codes is determined as one code word group.

[0109] The present application also provides an electronic device. Referring to FIG. 1, which shows a structural schematic diagram of an electronic device suitable for implementing the electronic device in the present application. The electronic device in the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, and the like. Figure 8 The electronic device shown in FIG. 1 is only an example and should not impose any limitation on the functions and use range of the present application. Figure 8 The electronic device shown in FIG. 1 is only an example and should not impose any limitation on the functions and use range of the present application.

[0110] As shown in FIG. 1, the electronic device includes a processor 101, a memory 102, a communication interface 103, and a power supply 104. Figure 8As shown, the electronic device can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. In a state in which the electronic device is powered on, various programs and data required for operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0111] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a memory card, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device having various devices is shown, but it is understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0112] The embodiment of the present application further provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the data file rare character garbled code processing methods provided by the embodiments of the present application.

[0113] The embodiment of the present application further provides a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data file rare character garbled code processing methods provided by the embodiments of the present application.

[0114] In addition, it should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0115] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.

[0116] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially.

[0117] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A method for processing garbled characters in a data file, characterized in that, include: The location of each Chinese character in the data file is determined, and the character encoding of each Chinese character is extracted based on the location. The character encoding of each Chinese character is queried. The query process includes: taking each byte as the current byte in the order of the bytes included in the character encoding, determining the encoding range that the byte group consisting of the current byte and the bytes before the current byte falls into, and if the encoding range is a garbled range, then the Chinese character is determined to be garbled, the bytes before the current byte are extracted, and the process of taking each byte as the current byte in sequence is stopped. A series of consecutive garbled characters is identified as a garbled word group. Using an association rule mining algorithm, each extracted byte of each garbled word group is processed along with high-frequency rare words included in a preset high-frequency rare word library to obtain rare words corresponding to the garbled word group. Based on the position of the garbled word group, the garbled word group is replaced with the rare words corresponding to the garbled word group.

2. The method for processing garbled characters in data files according to claim 1, characterized in that, The process of extracting each byte from each garbled word group and processing the high-frequency rare words included in the preset high-frequency rare word library includes: The bytes extracted from each garbled word group that fall within the range of rare characters are processed along with the high-frequency rare characters included in the preset high-frequency rare character word library.

3. The method for handling garbled characters in data files according to claim 1, characterized in that, The association rule mining algorithm is the Apriori algorithm. The algorithm is used to process each byte extracted from each garbled word group and the high-frequency rare words included in a preset high-frequency rare word library to obtain the rare words corresponding to the garbled word group, including: The Apriori algorithm is used to generate multiple optional rare character combinations corresponding to each extracted byte of each garbled word group. Each rare character in the optional rare character combination is a rare character in a preset high-frequency rare character dictionary. The Apriori algorithm is used to determine the support level of each of the optional rare character combinations, and optional rare character combinations with support levels below a threshold are discarded. The Apriori algorithm is used to generate association rules based on the remaining optional rare character combinations; Using the Apriori algorithm based on the association rules, determine the confidence level of each of the remaining optional rare character combinations; The Apriori algorithm is used to determine the highest confidence-based combination of rare characters as the rare characters corresponding to the garbled word group.

4. The method for handling garbled characters in data files according to claim 1, characterized in that, After replacing the garbled words with uncommon characters corresponding to the garbled words, the process returns to the position of each Chinese character included in the location data file, and extracts the character encoding of each Chinese character based on the position, until no Chinese character is identified as garbled.

5. The method for handling garbled characters in a data file according to claim 1, characterized in that, The step of identifying consecutively arranged garbled characters as a garbled word group includes: A sequence of consecutive garbled characters is defined as a garbled word group. If there are garbled characters that are not consecutively arranged with other garbled characters, then the garbled characters that are not consecutively arranged with other garbled characters are defined as a garbled word group.

6. A device for processing garbled characters in data files, characterized in that, include: The uncommon character garbled text detection module is used to locate the position of each Chinese character in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character. The query processing includes: taking each byte as the current byte in the order of the bytes included in the character code, determining the encoding range that the byte group composed of the current byte and the bytes before the current byte falls into, and if the encoding range falls into the garbled text range, then the Chinese character is determined to be garbled text, and the bytes before the current byte are extracted and the sequential taking of each byte as the current byte is stopped. The uncommon character garbled text recovery module is used to identify consecutively arranged garbled text as a garbled word group. Using an association rule mining algorithm, it processes each extracted byte of each garbled word group and high-frequency uncommon words included in a preset high-frequency uncommon word library to obtain the uncommon words corresponding to the garbled word group. Based on the position of the garbled word group, it replaces the garbled word group with the uncommon words corresponding to the garbled word group.

7. The device for processing garbled characters in data files according to claim 6, characterized in that, The rare character garbled text recovery module processes each extracted byte of each garbled word group and the high-frequency rare characters and words included in the preset high-frequency rare character and word library, specifically as follows: The bytes extracted from each garbled word group that fall within the range of rare characters are processed along with the high-frequency rare characters included in the preset high-frequency rare character word library.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the method for processing garbled characters in a data file as described in any one of claims 1 to 5.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the method for processing garbled characters in a data file as described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the method for handling garbled characters in a data file as described in any one of claims 1 to 5.

Images