Method for processing uncommon character messy codes in data file and related device

Abstract

The invention discloses a method and a related device for processing uncommon character messy codes in a data file, and relates to the field of data recovery, the method comprises the following steps: coding characters of Chinese characters, and carrying out query processing on the character codes of the Chinese characters, the query processing comprises the following steps: according to an arrangement sequence of bytes included in the character codes, sequentially taking the bytes as current bytes, and carrying out query processing on the current bytes; determining a coding range in which a byte group formed by the current byte and bytes before the current byte falls, and if the coding range falls into a messy code range, determining that the Chinese character is a messy code, and extracting each byte before the current byte; determining the continuously arranged messy codes as messy code phrases, processing each byte extracted from each messy code phrase and high-frequency rarely-used words included in a preset high-frequency rarely-used word library by using an association rule mining algorithm to obtain rarely-used words corresponding to the messy code phrases, and replacing the messy code phrases with uncommon characters and words corresponding to the messy code phrases. According to the invention, the use of the data file is prevented from being influenced by the messy codes.

Description

Technical Field

[0001] The present application relates to the technical field of data recovery, and in particular to a method and a related device for processing rare characters and garbled characters in a data file. Background Art

[0002] With the popularization of electronic office, the use of data files such as spreadsheets is becoming more and more popular, and the types and contents of data in data files are also increasing.

[0003] In data files, some uncommon characters may appear, and these uncommon characters can easily become garbled characters during storage and transmission, causing the use of data files to be affected by these garbled characters. Summary of the invention

[0004] In view of the above problems, the present application provides a method and a related device for processing garbled characters in a data file, so as to achieve the purpose of restoring garbled characters to garbled characters. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for processing garbled characters in a data file, comprising:

[0006] Locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character, the query processing comprising: according to the arrangement order of bytes included in the character code, take each byte as the current byte in turn, determine the encoding range that the byte group consisting of the current byte and the byte before the current byte falls into, if the encoding range falls into the garbled code range, determine that the Chinese character is garbled code, extract each byte before the current byte and stop taking each byte as the current byte in turn;

[0007] The continuously arranged garbled characters are determined as a garbled character phrase, and the extracted bytes of each garbled character phrase and the high-frequency rare words included in a preset high-frequency rare word library are processed using an association rule mining algorithm to obtain the rare words corresponding to the garbled character phrase, and based on the position of the garbled character phrase, the garbled character phrase is replaced with the rare words corresponding to the garbled character phrase.

[0008] In a possible implementation, the processing of each byte extracted from each garbled word group and the high-frequency rare words included in the preset high-frequency rare word word library includes:

[0009] The bytes extracted from each garbled word group that fall within the range of rare characters and the high-frequency rare characters included in the preset high-frequency rare character word library are processed.

[0010] In a possible implementation, the association rule mining algorithm is an Apriori algorithm, and the association rule mining algorithm is used to process the extracted bytes of each garbled word group and the high-frequency rare words included in the preset high-frequency rare word library to obtain the rare words corresponding to the garbled word group, including:

[0011] Apriori algorithm is used to generate a plurality of optional rare character combinations corresponding to each byte extracted from each garbled word group, wherein each rare character in the optional rare character combination is a rare character in a preset high-frequency rare character word library;

[0012] Determine the support of each optional rare character combination by using the Apriori algorithm, and discard the optional rare character combinations whose support is lower than a threshold;

[0013] Generate association rules based on the remaining optional rare character combinations using the Apriori algorithm;

[0014] Determine the confidence of each remaining optional rare character combination based on the association rule using the Apriori algorithm;

[0015] The Apriori algorithm is used to determine the optional rare character combination with the highest confidence as the rare character word corresponding to the garbled word group.

[0016] In one possible implementation, after replacing the garbled phrase with the uncommon words corresponding to the garbled phrase, the step of returning to execute the position of each Chinese character included in the positioning data file and extracting the character encoding of each Chinese character based on the position is performed until no Chinese characters are determined to be garbled.

[0017] In a possible implementation, determining the continuously arranged garbled characters as a garbled character phrase includes:

[0018] Determine the consecutively arranged garbled characters as a garbled character phrase;

[0019] If there are garbled codes that are not arranged continuously with other garbled codes, the garbled codes that are not arranged continuously with other garbled codes are determined as a garbled code phrase.

[0020] A second aspect of the present application provides a device for processing rare characters and garbled characters in a data file, comprising:

[0021] The module for checking the garbled codes of rare characters is used to locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character, wherein the query processing includes: according to the arrangement order of the bytes included in the character code, taking each byte as the current byte in turn, determining the encoding range in which the byte group consisting of the current byte and the byte before the current byte falls, and if the encoding range falls into the garbled code range, determining that the Chinese character is garbled code, extracting each byte before the current byte and stopping taking each byte as the current byte in turn;

[0022] The module for recovering rare characters and garbled codes is used to determine the continuously arranged garbled codes as a garbled code phrase, and use the association rule mining algorithm to process the extracted bytes of each garbled code phrase and the high-frequency rare characters included in the preset high-frequency rare characters word library to obtain the rare characters corresponding to the garbled code phrase, and replace the garbled code phrase with the rare characters corresponding to the garbled code phrase based on the position of the garbled code phrase.

[0023] In a possible implementation, the uncommon character garbled code recovery module processes each byte extracted from each garbled code phrase and the high-frequency uncommon characters included in the preset high-frequency uncommon character word library, and is specifically configured as follows:

[0024] The bytes extracted from each garbled word group that fall within the range of rare characters and the high-frequency rare characters included in the preset high-frequency rare character word library are processed.

[0025] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for processing rare characters and garbled characters in a data file according to the first aspect or any implementation method of the first aspect.

[0026] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0027] The memory is used to store computer programs;

[0028] The processor is used to execute the computer program so that the electronic device can implement the method for processing rare characters and garbled characters in a data file in the first aspect or any implementation method of the first aspect.

[0029] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the method for processing rare and garbled characters in a data file in the first aspect or any implementation method of the first aspect.

[0030] By means of the above technical scheme, the method and related device for processing rare characters and garbled codes in a data file provided by the present application can locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character, wherein the query processing includes: according to the arrangement order of the bytes included in the character code, taking each byte as the current byte in turn, determining the encoding range in which the byte group consisting of the current byte and the byte before the current byte falls, if the encoding range falls into the garbled code range, determining that the Chinese character is garbled code, extracting each byte before the current byte and stopping taking each byte as the current byte in turn; determining the continuously arranged garbled codes as a garbled code phrase, using an association rule mining algorithm, processing each byte extracted from each garbled code phrase and the high-frequency rare characters and words included in a preset high-frequency rare character word library, obtaining the rare characters and words corresponding to the garbled code phrase, and replacing the garbled code phrase with the rare characters and words corresponding to the garbled code phrase based on the position of the garbled code phrase. The present application can identify garbled characters based on character encoding, and extract byte groups that do not fall into the garbled character range in the encoding of the garbled characters, and use the association rule mining algorithm and a preset high-frequency rare word library to obtain rare words corresponding to the garbled character phrases, and then use the garbled character phrases to replace the garbled character phrases with the rare words corresponding to the garbled character phrases. In this way, the present application can restore the garbled characters to rare words, avoiding the use of data files being affected by these garbled characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0032] Figure 1 A schematic diagram of a system architecture provided for this application;

[0033] Figure 2 A schematic diagram of an optional hardware structure of a terminal provided in this application;

[0034] Figure 3 A schematic diagram of the structure of the server provided for this application;

[0035] Figure 4 A flowchart of a method for processing garbled characters in a data file provided by an embodiment of the present application;

[0036] Figure 5 This is a schematic diagram of commonly used Chinese character encoding;

[0037] Figure 6 Schematic diagram of coding for rare characters;

[0038] Figure 7 A schematic diagram of the structure of a device for processing rare characters and garbled characters in a data file provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0041] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0042] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0043] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0044] Among them, an application with a data file recovery function can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can obtain the data file selected by the user on the data file upload interface and send the above data file to the server 200. The server 200 can obtain the processing result (the data file after replacing the garbled characters with rare characters) based on the received data file, and return the processing result to the terminal 100.

[0045] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the obtained data file by itself without the need for the cooperation of the server, and the embodiments of the present application are not limited thereto.

[0046] Next describe Figure 1 The product form of the mid-terminal 100;

[0047] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0048] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0049] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0050] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0051] Among them, the input device 132 can receive input data and the like.

[0052] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display a data file upload interface, processing results, etc.

[0053] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0054] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0055] Among them, the memory 120 can be used to store software codes related to the method for processing rare and garbled characters in data files, the processor 170 can execute the steps of the method for processing rare and garbled characters in data files, and can also dispatch other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0056] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0057] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0058] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0059] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0060] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0061] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0062] Next describe Figure 1 The product form of the server 200;

[0063] Figure 3 A schematic diagram of the structure of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0064] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0065] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0066] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0067] Among them, the memory 204 can be used to store software codes related to the method for processing rare and garbled characters in data files, the processor 202 can execute the steps of the method for processing rare and garbled characters in the data files of the chip, and can also schedule other units to realize corresponding functions.

[0068] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0069] In order to solve the above problems, the embodiment of the present application provides a method for processing uncommon characters and garbled codes in a data file. The method for processing uncommon characters and garbled codes in a data file of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0070] Reference Figure 4 , Figure 4 The process diagram of a method for processing rare characters and garbled characters in a data file provided in an embodiment of the present application is as follows: Figure 4 As shown, a data processing method provided in an embodiment of the present application may include steps 401 to 402, and these steps are described in detail below.

[0071] 401. Locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character. The query processing includes: according to the arrangement order of the bytes included in the character code, each byte is used as the current byte in turn, and the encoding range that the byte group composed of the current byte and the byte before the current byte falls into is determined. If the encoding range falls into the garbled code range, it is determined that the Chinese character is garbled, and the bytes before the current byte are extracted and the bytes before the current byte are stopped.

[0072] Data files are files used to store data. There are many types of data files, such as tables and database files.

[0073] For some data files (such as a financial institution's business outlet information table), since the format of the data they carry is relatively fixed, the places in the data file that are Chinese characters are pre-set, and this application can locate the positions of each Chinese character included in the data file according to the pre-set Chinese character positions. For example: the position, content, and format of each field in a financial institution's business outlet information table are fixed, where the fourth column is the name of the legal person of the business outlet, and the legal person's name is Chinese characters, then this application can determine that the data in each cell of the fourth column is Chinese characters.

[0074] In another optional embodiment, the present application can also obtain the position information of Chinese characters from the data file or outside the data file, for example, the position information of Chinese characters is carried in the data file, or the position information of Chinese characters in the data file is stored in another file.

[0075] Character encoding, also known as character set code, is the encoding of characters in a character set into an object in a character set so that text can be stored in a computer and transmitted through a communication network. A character set is a mapping table that corresponds one character to a character encoding. Common character sets include ASCII, Unicode, etc.

[0076] After determining the position of the Chinese character, the character code of the Chinese character can be extracted from the corresponding position. It should be noted that when the character code is extracted, since the query processing has not been performed, it is not known whether the Chinese character at the position is displayed normally. If it is displayed normally, the extracted character code will be the code of the Chinese character. If it is displayed as garbled, the extracted character code will be the code of the garbled code. In order to determine whether the Chinese character is garbled, this application can query the character code of each Chinese character separately.

[0077] Taking Unicode as an example, Figure 5 As shown in the figure, the Unicode encoding range of commonly used Chinese characters is from \u4e00 to \u9FFF. Figure 6 As shown, the Unicode encoding range of uncommon Chinese characters (i.e. rare characters) is from \u20000 to \u2A6DF. The first letter u in the above encoding range represents the Unicode encoding, and the bytes included in the character encoding using Unicode encoding are the content after the letter u, which is generally four bytes or five bytes. For example, the character encoding of the above common Chinese character Unicode encoding range includes four bytes, and the encoding range of rare characters includes five bytes.

[0078] For ease of understanding, the following example uses the extracted character code u2A6FF as an example:

[0079] First, according to the order of the five bytes 2, A, 6, F, F included in u2A6FF, take 2 as the current byte and determine The coding range that falls into. In the Unicode encoding range of Chinese, the next Unicode code A is determined as the current byte, and the byte group consisting of A and the 2 before A is determined. The coding range it falls into. It is still within the Chinese range, so the next Unicode code 6 is determined as the current byte, and the byte group consisting of 6 and the 2A before 6 is determined. The encoding range it falls into is repeated in this way, and the loop is executed until the encoding range that the byte group falls into is not within the Chinese range or there is no corresponding encoding. In this example, due to It is no longer within the Chinese range, that is, the Chinese characters are determined to be garbled, and the bytes before the current byte are extracted, that is, .

[0080] 402. Determine the continuously arranged garbled characters as a garbled character phrase, use an association rule mining algorithm to process the extracted bytes of each garbled character phrase and the high-frequency rare words included in a preset high-frequency rare word database, obtain the rare words corresponding to the garbled character phrase, and replace the garbled character phrase with the rare words corresponding to the garbled character phrase based on the position of the garbled character phrase.

[0081] Generally, after the garbled characters appear, they appear continuously, so the present application can determine the continuously arranged garbled characters as a garbled character phrase. Since the location of the multiple continuously arranged garbled characters should be the continuously arranged Chinese characters, and the continuously arranged Chinese characters have semantic continuity, such as a Chinese word or even a Chinese short sentence. Therefore, if the analysis is performed based on the garbled character phrase, not only can the corresponding uncommon characters be tried to be found based on the character encoding of the garbled characters themselves, but also the corresponding uncommon characters can be tried to be found based on the relationship between the character encodings of the multiple garbled characters.

[0082] Optionally, determining the continuously arranged garbled characters as a garbled character phrase in step 402 may include:

[0083] The garbled codes that are arranged continuously are determined as a garbled code phrase. If there are garbled codes that are not arranged continuously with other garbled codes, the garbled codes that are not arranged continuously with other garbled codes are determined as a garbled code phrase.

[0084] If there are garbled codes that are not arranged continuously with other garbled codes, the garbled codes that are not arranged continuously with other garbled codes are determined as a garbled code phrase, and an attempt is made to find a corresponding uncommon character.

[0085] In practical applications, a preset high-frequency rare word word library can be established according to the actual business or industry. For example, for the financial industry, the high-frequency rare words commonly used in the financial industry can be combined into a preset high-frequency rare word word library. Of course, in addition to high-frequency rare words, the preset high-frequency rare word word library can also include words composed of multiple high-frequency rare words. In this application, words composed of multiple rare words are called rare word words.

[0086] Specifically, step 402 processes each byte extracted from each garbled word group and the high-frequency rare words included in the preset high-frequency rare word word library, which may include:

[0087] The bytes extracted from each garbled word group that fall within the range of rare characters and the high-frequency rare characters included in the preset high-frequency rare character word library are processed.

[0088] If the Chinese character does not fall into the range of rare characters, it is possible that the Chinese character is not a rare character and no processing is required.

[0089] Optionally, the association rule mining algorithm is the Apriori algorithm. The Apriori algorithm can use a layer-by-layer search iterative method to find the relationship between item sets in the database to form rules, and its process consists of connection (matrix-like operation) and pruning (removing unnecessary intermediate results).

[0090] Step 402 uses an association rule mining algorithm to process the extracted bytes of each garbled word group and the high-frequency rare words included in the preset high-frequency rare word word library to obtain the rare words corresponding to the garbled word group, which may specifically include:

[0091] Apriori algorithm is used to generate multiple optional rare character combinations corresponding to each byte extracted from each garbled word group, wherein each rare character in the optional rare character combination is a rare character in a preset high-frequency rare character word library;

[0092] The Apriori algorithm is used to determine the support of each optional rare character combination, and the optional rare character combinations with support below the threshold are discarded;

[0093] Generate association rules based on the remaining optional rare word combinations using the Apriori algorithm;

[0094] Using the Apriori algorithm based on association rules, the confidence of each remaining optional rare word combination is determined;

[0095] The Apriori algorithm is used to determine the optional rare character combination with the highest confidence as the rare character word corresponding to the garbled phrase.

[0096] Optionally, in another optional embodiment of the present invention, after replacing the garbled phrase with the uncommon words corresponding to the garbled phrase, the step of returning to execute the position of each Chinese character included in the positioning data file and extracting the character encoding of each Chinese character based on the position can be performed until no Chinese characters are determined to be garbled.

[0097] So, if you execute once Figure 1 If the method shown cannot restore all the garbled characters, or new garbled characters are introduced, you can continue to execute Figure 1 The method shown is repeated until there is no garbled code in the data file.

[0098] A method for processing rare characters and garbled characters in a data file provided by an embodiment of the present application is introduced above. A device for executing the above method for processing rare characters and garbled characters in a data file will be introduced below.

[0099] See also Figure 7 , Figure 7 The present invention provides a schematic diagram of a device for processing rare characters and garbled characters in a data file. Figure 7 As shown, the device for processing rare characters and garbled codes in the data file includes:

[0100] The uncommon character garbled code checking module 701 is used to locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character. The query processing includes: according to the arrangement order of the bytes included in the character code, each byte is used as the current byte in turn, and the encoding range that the byte group composed of the current byte and the byte before the current byte falls into is determined. If the encoding range falls into the garbled code range, it is determined that the Chinese character is garbled, and each byte before the current byte is extracted and each byte is stopped from being used as the current byte in turn;

[0101] The module 702 for recovering the garbled characters is used to determine the continuously arranged garbled characters as a garbled character phrase, and to process the extracted bytes of each garbled character phrase and the high-frequency garbled characters included in the preset high-frequency garbled character word library by using the association rule mining algorithm to obtain the garbled characters corresponding to the garbled character phrase, and to replace the garbled character phrase with the garbled characters corresponding to the garbled character phrase based on the position of the garbled character phrase.

[0102] Optionally, the module for recovering the rare characters from garbled codes processes the extracted bytes of each garbled code phrase and the high-frequency rare characters and words included in the preset high-frequency rare characters word library, and is specifically configured as follows:

[0103] The bytes extracted from each garbled word group that fall within the range of rare characters and the high-frequency rare characters included in the preset high-frequency rare character word library are processed.

[0104] Optionally, the association rule mining algorithm is an Apriori algorithm, and the uncommon word garbled code recovery module uses the association rule mining algorithm to process the extracted bytes of each garbled code phrase and the high-frequency uncommon words included in the preset high-frequency uncommon word library to obtain the uncommon words corresponding to the garbled code phrase, which is specifically set as:

[0105] Apriori algorithm is used to generate multiple optional rare character combinations corresponding to each byte extracted from each garbled phrase, each rare character in the optional rare character combination is a rare character in a preset high-frequency rare character vocabulary, Apriori algorithm is used to determine the support of each optional rare character combination, and the optional rare character combinations with support below a threshold are discarded, Apriori algorithm is used to generate association rules based on the remaining optional rare character combinations, Apriori algorithm is used to determine the confidence of each remaining optional rare character combination based on the association rules, and Apriori algorithm is used to determine the optional rare character combination with the highest confidence as the rare character word corresponding to the garbled phrase.

[0106] Optionally, after the uncommon character garbled code recovery module replaces the garbled code phrase with the uncommon characters corresponding to the garbled code phrase, the uncommon character garbled code checking module is triggered to process until no Chinese characters are determined to be garbled codes.

[0107] Optionally, the uncommon character garbled code recovery module determines the continuously arranged garbled codes as a garbled code phrase, and is specifically set as:

[0108] The garbled codes that are arranged continuously are determined as a garbled code phrase. If there are garbled codes that are not arranged continuously with other garbled codes, the garbled codes that are not arranged continuously with other garbled codes are determined as a garbled code phrase.

[0109] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0110] like Figure 8As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0111] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0112] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements a method for processing rare characters and garbled characters in any data file provided in an embodiment of the present application.

[0113] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement a method for processing rare characters and garbled characters in any data file provided in an embodiment of the present application.

[0114] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0115] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0116] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0117] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for processing rare characters and garbled characters in a data file, characterized in that: include: Locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character, the query processing comprising: according to the arrangement order of the bytes included in the character code, take each byte as the current byte in turn, determine the encoding range that the byte group consisting of the current byte and the byte before the current byte falls into, if the encoding range falls into the garbled code range, determine that the Chinese character is garbled code, extract each byte before the current byte and stop taking each byte as the current byte in turn; The continuously arranged garbled characters are determined as a garbled character phrase, and the extracted bytes of each garbled character phrase and the high-frequency rare words included in a preset high-frequency rare word library are processed using an association rule mining algorithm to obtain the rare words corresponding to the garbled character phrase, and based on the position of the garbled character phrase, the garbled character phrase is replaced with the rare words corresponding to the garbled character phrase.

2. The method for processing rare characters and garbled characters in a data file according to claim 1, characterized in that: The step of processing each byte extracted from each garbled word group and the high-frequency rare words included in the preset high-frequency rare word word library comprises: The bytes extracted from each garbled word group that fall within the range of rare characters and the high-frequency rare characters included in the preset high-frequency rare character word library are processed.

3. The method for processing rare characters and garbled characters in a data file according to claim 1, characterized in that: The association rule mining algorithm is an Apriori algorithm. The association rule mining algorithm is used to process each byte extracted from each garbled word group and the high-frequency rare words included in the preset high-frequency rare word library to obtain the rare words corresponding to the garbled word group, including: Apriori algorithm is used to generate a plurality of optional rare character combinations corresponding to each byte extracted from each garbled word group, wherein each rare character in the optional rare character combination is a rare character in a preset high-frequency rare character word library; Determine the support of each optional rare character combination by using the Apriori algorithm, and discard the optional rare character combinations whose support is lower than a threshold; Generate association rules based on the remaining optional rare character combinations using the Apriori algorithm; Determine the confidence of each remaining optional rare character combination based on the association rule using the Apriori algorithm; The Apriori algorithm is used to determine the optional rare character combination with the highest confidence as the rare character word corresponding to the garbled word group.

4. The method for processing rare characters and garbled characters in a data file according to claim 1, characterized in that: After replacing the garbled word group with the rare words corresponding to the garbled word group, return to execute the position of each Chinese character included in the positioning data file, and extract the character code of each Chinese character based on the position until no Chinese character is determined to be garbled.

5. The method for processing rare characters and garbled characters in a data file according to claim 1, characterized in that: The step of determining the continuously arranged garbled characters as a garbled character phrase comprises: The garbled codes that are arranged continuously are determined as a garbled code phrase. If there are garbled codes that are not arranged continuously with other garbled codes, the garbled codes that are not arranged continuously with other garbled codes are determined as a garbled code phrase.

6. A device for processing rare characters and garbled characters in a data file, characterized in that: include: The module for checking the garbled codes of rare characters is used to locate the position of each Chinese character included in the data file, extract the character code of each Chinese character based on the position, and perform query processing on the character code of each Chinese character, wherein the query processing includes: according to the arrangement order of the bytes included in the character code, taking each byte as the current byte in turn, determining the encoding range in which the byte group consisting of the current byte and the byte before the current byte falls, and if the encoding range falls into the garbled code range, determining that the Chinese character is garbled code, extracting each byte before the current byte and stopping taking each byte as the current byte in turn; The module for recovering rare characters and garbled codes is used to determine the continuously arranged garbled codes as a garbled code phrase, and use the association rule mining algorithm to process the extracted bytes of each garbled code phrase and the high-frequency rare characters included in the preset high-frequency rare characters word library to obtain the rare characters corresponding to the garbled code phrase, and replace the garbled code phrase with the rare characters corresponding to the garbled code phrase based on the position of the garbled code phrase.

7. The device for processing rare characters and garbled codes in a data file according to claim 6, characterized in that: The module for recovering the rare characters from garbled codes processes the extracted bytes of each garbled code phrase and the high-frequency rare characters included in the preset high-frequency rare characters word library, and is specifically configured as follows: The bytes extracted from each garbled word group that fall within the range of rare characters and the high-frequency rare characters included in the preset high-frequency rare character word library are processed.

8. A computer program product, characterized in that It includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for processing rare characters and garbled characters in a data file as described in any one of claims 1 to 5.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the method for processing rare characters and garbled characters in a data file as described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the method for processing rare characters and garbled characters in a data file as described in any one of claims 1 to 5.

Images