Method, device, storage medium and electronic device for detecting file encoding

Abstract

The application discloses a method and device for detecting file encoding, a storage medium and an electronic device. The method comprises the following steps: receiving a file to be detected; obtaining a target byte of byte stream data corresponding to the file to be detected; judging whether the file to be detected is a standard encoding format type according to the target byte; in the case that the file to be detected is a non-standard encoding format type, dividing the byte stream data to which the file to be detected belongs uniformly to obtain a byte stream segment set, wherein the non-standard encoding format type contains multiple specified encoding formats; and determining the encoding format of the file to be detected from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment. The application solves the technical problem of low detection efficiency caused by the single encoding format type detection method in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the coding field, in particular to a method and device for detecting file coding, a storage medium and an electronic device. BACKGROUND

[0002] With the expansion of the application range of computer programs, more and more types of data need to be processed by programs. Some of these data will be stored in databases (such as oracle and mySql); some will be stored in cache middleware (such as redis and rabbitmq); and some will be directly stored in the form of text files on servers. The source platforms and generation methods of the files are different, so different files will have multiple encoding formats, such as UTF-8, GBK, and ISO-8859-1. When reading or sending the text files stored on the server, improper operation can easily cause the files to appear in a garbled manner, and when the garbled files are read again, there is a risk of loss of file information.

[0003] At present, in the method for detecting file encoding format, there are methods for detecting standard encoding files and methods for detecting non-standard encoding file formats, but all of them are single encoding format detection methods, which are low in detection efficiency.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a method and device for detecting file coding, a storage medium and an electronic device to at least solve the technical problem of low detection efficiency caused by the single encoding format detection method in the prior art.

[0006] According to an aspect of an embodiment of the present application, a method for detecting file coding is provided, comprising: receiving a file to be detected; obtaining a target byte of byte stream data corresponding to the file to be detected; determining whether the file to be detected is a standard encoding format type according to the target byte; in the case that the file to be detected is a non-standard encoding format type, dividing the byte stream data to which the file to be detected belongs uniformly to obtain a byte stream segment set, wherein the non-standard encoding format type contains multiple specified encoding formats; and determining the encoding format of the file to be detected from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

[0007] Optionally, determining whether the file to be detected is a standard encoding format type according to the target byte comprises: determining whether the target byte has a byte order mark (BOM); in the case that the target byte has the BOM, determining that the encoding format type of the file to be detected is the standard encoding format type; and in the case that the target byte does not have the BOM, determining that the encoding format type of the file to be detected is the non-standard encoding format type.

[0008] Optionally, the byte stream data to which the file to be detected belongs is uniformly divided, comprising: determining the total length of the byte stream data; determining the unit length according to the total length and a predetermined number, wherein the sum of the predetermined number of unit lengths is equal to the total length; uniformly dividing the byte stream data according to the unit length.

[0009] Optionally, the encoding format of the file to be detected is determined from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, comprising: detecting the encoding format of each byte stream segment; determining the number of byte stream segments of the garbled format in the byte stream segment set; comparing the number of byte stream segments with a preset threshold; determining whether the file to be detected is of the garbled format according to the comparison result.

[0010] Optionally, the encoding format of the file to be detected is determined from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, comprising: determining the number of byte stream segments of the garbled format in the byte stream segment set; comparing the number of byte stream segments with a preset threshold; determining whether the file to be detected is of the garbled format according to the comparison result.

[0011] Optionally, the encoding format of the file to be detected is determined from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, comprising: determining the number of byte stream segments of the garbled format in the byte stream segment set; comparing the number of byte stream segments with a preset threshold; determining whether the file to be detected is of the garbled format according to the comparison result.

[0012] Optionally, the encoding format of the file to be detected is determined from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, comprising: determining the number of byte stream segments of the garbled format in the byte stream segment set; comparing the number of byte stream segments with a preset threshold; determining whether the file to be detected is of the garbled format according to the comparison result.

[0013] According to another aspect of the embodiment of the present application, a method for detecting the encoding of a file is also provided, comprising: a target device receiving a file to be detected; determining the encoding type of the file to be detected, the type comprising: a standard encoding format type and a non-standard encoding format type; determining a target detection method corresponding to the encoding type from the detection methods pre-stored in the target device, wherein the detection method is used to detect the encoding format of the file to be detected, and the detection methods pre-stored in the target device comprise: a first detection method corresponding to the standard encoding format type and a second detection method corresponding to the non-standard encoding format type; and determining the encoding format of the file to be detected by using the target detection method.

[0014] According to another aspect of the embodiments of the present application, a device for detecting file encoding is also provided, comprising: a receiving module configured to receive a file to be detected; a judging module configured to obtain a target byte of byte stream data corresponding to the file to be detected, and determine whether the file to be detected is of a standard encoding format type according to the target byte; a dividing module configured to, if the file to be detected is of a non-standard encoding format type, divide the byte stream data to which the file to be detected belongs evenly to obtain a byte stream segment set, wherein the non-standard encoding format type comprises a plurality of specified encoding formats; and a determining module configured to determine an encoding format of the file to be detected from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, comprising: a storage medium comprising a stored program, wherein the program, when executed, controls a device in which the storage medium is located to perform any one of the methods for detecting file encoding.

[0016] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any one of the methods for detecting file encoding.

[0017] In the embodiments of the present application, the corresponding detection method is determined according to the encoding type of the file to be detected, by receiving the file to be detected; obtaining a target byte of byte stream data corresponding to the file to be detected, and determining whether the file to be detected is of a standard encoding format type according to the target byte; if the file to be detected is of a non-standard encoding format type, dividing the byte stream data to which the file to be detected belongs evenly to obtain a byte stream segment set; and determining an encoding format of the file to be detected from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, so as to improve the detection efficiency, thereby realizing the technical effect of reducing the risk of file loss caused by garbled code when reading or sending a file, and further solving the technical problem of low detection efficiency caused by the single encoding format type detection method in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description, serve to explain the present application. In the drawings:

[0019] Figure 1 is a flowchart of a method for detecting file encoding according to an embodiment of the present application;

[0020] Figure 2is a schematic diagram of another optional method for detecting file encoding according to an embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of a device for detecting file encoding according to an embodiment of the present application;

[0022] Figure 4 is a schematic block diagram of an electronic device 400 according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In order to facilitate better understanding of the present application by those skilled in the art, the technical terms or part of the nouns that may be involved in the present application will be explained:

[0026] BOM (byte-order mark), i.e. byte order mark, is a special mark inserted at the beginning of data encoded in UTF-8, UTF16 or UTF-32, used to identify the encoding type of the data. A character U+FEFF is defined in the encoding table, which does not have an actual corresponding symbol. If the character U+FEFF appears at the beginning of the byte stream, it is used to identify the byte order of the byte stream, whether the high bit is in front or the low bit is in front.

[0027] According to an embodiment of the present application, a method for detecting file encoding is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] Figure 1 A method for detecting file encoding is provided according to an embodiment of the present application, as shown in the accompanying drawings, the method comprises the following steps: Figure 1

[0029] Step S102, receiving a file to be detected;

[0030] Step S104, obtaining a target byte of byte stream data corresponding to the file to be detected, and determining whether the file to be detected is a standard encoding format type according to the target byte;

[0031] It should be noted that the standard encoding format type refers to that the file to be detected contains only one encoding format.

[0032] Step S106, in the case that the file to be detected is a non-standard encoding format type, dividing the byte stream data to which the file to be detected belongs uniformly to obtain a byte stream segment set, wherein the non-standard encoding format type contains multiple specified encoding formats;

[0033] It should be noted that the non-standard encoding format type refers to that the file to be detected contains multiple encoding formats, wherein the encoding formats include but are not limited to Unicode encoding format, Unicode big endian encoding format and UTF-8 encoding format.

[0034] Step S108, determining the encoding format of the file to be detected from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

[0035] It can be understood that the file of the non-standard encoding format type has multiple encoding formats, that is, the encoding format set corresponding to the non-standard encoding format type includes but is not limited to Unicode encoding format, Unicode big endian encoding format and UTF-8 encoding format.

[0036] ​In the embodiment of the present application, the corresponding detection method is determined according to the encoding type of the file to be detected, and the file to be detected is received. The target byte of the byte stream data corresponding to the file to be detected is obtained, and it is judged whether the file to be detected is of a standard encoding format type according to the target byte. In the case that the file to be detected is of a non-standard encoding format type, the byte stream data to which the file to be detected belongs is uniformly divided to obtain a byte stream segment set. The encoding format of the file to be detected is determined from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment. The purpose of improving the detection efficiency is achieved, thereby realizing the technical effect of reducing the risk of file loss caused by garbled code when reading or sending files, and further solving the technical problem of low detection efficiency caused by the single encoding format type detection method in the prior art.

[0037] In an example embodiment of the present application, it is judged whether the file to be detected is of a standard encoding format type according to the target byte, including: judging whether the target byte has a byte order mark (BOM); in the case that the target byte has the BOM, determining that the encoding format type of the file to be detected is a standard encoding format type; in the case that the target byte does not have the BOM, determining that the encoding format type of the file to be detected is a non-standard encoding format type.

[0038] For example, in the case that the byte order mark is FF FE, it is determined that the encoding format of the file to be detected is Unicode; in the case that the byte order mark is FE FF, it is determined that the encoding format of the file to be detected is Unicode big endian; in the case that the byte order mark is EF BB BF, it is determined that the encoding format of the file to be detected is UTF-8.

[0039] As an optional implementation, the byte stream data to which the file to be detected belongs is uniformly divided, including: determining the total length of the byte stream data; determining the unit length according to the total length and a predetermined number, wherein the sum of the predetermined number of unit lengths is equal to the total length; uniformly dividing the byte stream data according to the unit length.

[0040] For example, assuming that the length of the byte stream data is 10, the byte stream data is uniformly divided into 10 segments, wherein the greater the division number, the more accurate the finally determined encoding format.

[0041] In some optional embodiments of the present application, the encoding format of the file to be detected is determined from the encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, including: detecting the encoding format of each byte stream segment; determining the number of byte stream segments belonging to garbled code format in the byte stream segment set; comparing the number of byte stream segments with a preset threshold; judging whether the file to be detected is of garbled code format according to the comparison result.

[0042] In an example embodiment of the present application, determining whether the to-be-detected file is in a random code format according to the comparison result comprises: determining that the to-be-detected file is in a random code format when the number of byte stream segments is greater than a preset threshold; determining the number of byte stream segments in other encoding formats in the byte stream segment set when the number of byte stream segments is less than the preset threshold; and determining the encoding format of the to-be-detected file according to the number of byte stream segments in other encoding formats, wherein the other encoding formats are encoding formats other than the random code format.

[0043] For example, assuming that the preset threshold is 50 and the number of byte stream segments in the random code format is 60, the number of byte stream segments in the random code format is greater than the preset threshold, and thus the to-be-detected file is in the random code format; assuming that the preset threshold is 50 and the number of byte stream segments in the random code format is 40, the number of byte stream segments in the random code format is less than the preset threshold, and thus the to-be-detected file is not in the random code format.

[0044] Optionally, determining the encoding format of the to-be-detected file according to the number of byte stream segments in other encoding formats comprises: determining the byte stream segments in the encoding format with the largest number in the byte stream segment set, and taking the encoding format with the largest number as the encoding format of the to-be-detected file.

[0045] In some optional embodiments of the present application, determining the byte stream segments in the encoding format with the largest number in the byte stream segment set comprises: determining the number of byte stream segments in other encoding formats in the byte stream segment set when the number of byte stream segments in the random code format is the largest and the number of byte stream segments in the random code format does not exceed the preset threshold; and determining the byte stream segments in the encoding format with the largest number in the byte stream segment set except the random code format.

[0046] For example, assuming that the preset threshold is 50, the number of byte stream segments in the random code format is 40, the number of byte stream segments in the Unicode encoding format is 30, the number of byte stream segments in the Unicode big endian encoding format is 20, and the number of byte stream segments in the UTF-8 encoding format is 10, the number of byte stream segments in the random code format is the largest but less than the preset threshold, and thus the encoding format of the to-be-detected file is the encoding format with the largest number of byte stream segments except the random code format. It can be understood that the encoding format of the to-be-detected file is the Unicode encoding format.

[0047] Figure 2 is a schematic diagram of another optional method for detecting the encoding of a file according to an embodiment of the present application, as shown in Figure 2 The method comprises the following steps:

[0048] Step S202, the target device receives a to-be-detected file;

[0049] Step S204, determine the encoding type of the file to be detected, including: standard encoding format type and non-standard encoding format type;

[0050] Step S206, determine the target detection method corresponding to the encoding type from the detection method pre-stored in the target device, wherein the detection method is used to detect the encoding format of the file to be detected, and the detection method pre-stored in the target device includes: the first detection method corresponding to the standard encoding format type and the second detection method corresponding to the non-standard encoding format type;

[0051] Step S208, determine the encoding format of the file to be detected by using the target detection method.

[0052] It is easy to note that the detection method can detect files of different encoding types, rather than only detecting a single encoding type.

[0053] In order to facilitate those skilled in the art to better understand the technical scheme of the present application, a specific embodiment will be described, and the method comprises the following steps:

[0054] (1) package the go code for detection;

[0055] (2) in the scene where it needs to be used, introduce the package;

[0056] (3) place the target file under a certain path;

[0057] (4) use the detection program to detect the file to be detected;

[0058] (5) if the target file is a standard encoding, the first three bytes can be directly judged to output the file format, wherein when the file to be detected is a standard encoding:

[0059] The data in the file is stored according to the given character set, and the encoding information is stored in the first three bytes of the file. The value of the first three bytes is judged to determine the format of the encoding;

[0060] If the first two bytes are FF FE, the encoding format of the file to be detected is Unicode;

[0061] If the first two bytes are FE FF, the encoding format of the file to be detected is Unicode big endian;

[0062] If the first three bytes are EF BB BF, the encoding format of the file to be detected is UTF-8;

[0063] It should be noted that if the first three bytes are -17, -69, and -65, the encoding format of the file to be detected is also UTF-8.

[0064] (6) If the target file is non-standard encoding, the improved detector function is used to identify the encoding format of the file to be detected, wherein the file to be detected is non-standard encoding or garbled code:

[0065] The data in the file may not be stored in the file according to the given character set, so the encoding information is not stored in the first byte of the file, and there may be data in the file in multiple encoding formats;

[0066] The byte stream data corresponding to the file to be detected is divided into n equal parts, n is less than or equal to the file length;

[0067] Detect each byte stream segment in the divided byte stream set;

[0068] The encoding format with the highest occurrence frequency is output as the encoding format of the file to be detected;

[0069] It should be noted that the larger n is, the higher the accuracy of the final encoding format.

[0070] Set the file garbled code threshold, when the number of byte stream segments corresponding to the garbled code format is greater than the threshold, the file to be detected is defined as a garbled code format. If the number of byte stream segments corresponding to the garbled code format is the largest and does not exceed the threshold, the encoding format of the file to be detected is defined as the encoding format with the largest number of byte stream segments other than the garbled code format.

[0071] It is easy to note that the detection method adopted by the present application can quickly and accurately identify the encoding format of the file to be detected, and is suitable for standard encoding, non-standard encoding and garbled code files. At the same time, the method is easy to deploy and has good implementability.

[0072] Figure 3 A device for detecting file encoding according to an embodiment of the present application, as shown in Figure 3 The device comprises:

[0073] The receiving module 30 is used to receive the file to be detected;

[0074] The judgment module 32 is used to obtain the target byte of the byte stream data corresponding to the file to be detected, and judge whether the file to be detected is a standard encoding format type according to the target byte;

[0075] The division module 34 is used to uniformly divide the byte stream data to which the file to be detected belongs to obtain a byte stream segment set in the case that the file to be detected is a non-standard encoding format type, wherein the non-standard encoding format type contains multiple specified encoding formats;

[0076] The determining module 36 is configured to determine the encoding format of the file to be detected from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

[0077] In the device, the receiving module 30 is configured to receive the file to be detected, the judging module 32 is configured to obtain a target byte of the byte stream data corresponding to the file to be detected, and determine whether the file to be detected is of a standard encoding format type according to the target byte, the dividing module 34 is configured to divide the byte stream data to which the file to be detected belongs evenly to obtain a byte stream segment set in a case where the file to be detected is of a non-standard encoding format type, and the non-standard encoding format type includes multiple specified encoding formats, and the determining module 36 is configured to determine the encoding format of the file to be detected from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment, so that the detection efficiency is improved, the technical effect of reducing the risk of file loss caused by garbled code when reading or sending a file is achieved, and the technical problem of low detection efficiency caused by the single encoding format type detection method in the prior art is solved.

[0078] In some embodiments of the present application, the judging module 32 is further configured to determine whether the target byte has a byte order mark (BOM), determine that the encoding format type of the file to be detected is a standard encoding format type in a case where the target byte has the BOM, and determine that the encoding format type of the file to be detected is a non-standard encoding format type in a case where the target byte does not have the BOM.

[0079] The dividing module 34 is further configured to determine a total length of the byte stream data, determine a unit length according to the total length and a predetermined number, wherein the sum of the predetermined number of unit lengths is equal to the total length, and evenly divide the byte stream data according to the unit length.

[0080] The determining module 36 is further configured to determine a byte stream segment of a most numerous encoding format in the byte stream segment set, and take the most numerous encoding format as the encoding format of the file to be detected.

[0081] The embodiments of the present application further provide a non-volatile storage medium, which includes a stored program, wherein the non-volatile storage medium controls a device in which the non-volatile storage medium is located to perform any one of the methods for detecting a file encoding when the program is running.

[0082] Specifically, the storage medium is configured to store program instructions of the following functions to implement the following functions:

[0083] The file to be detected is received, a target byte of byte stream data corresponding to the file to be detected is obtained, and whether the file to be detected is of a standard encoding format type is determined according to the target byte; in a case where the file to be detected is of a non-standard encoding format type, byte stream segments are obtained by uniformly dividing byte stream data to which the file to be detected belongs, wherein the non-standard encoding format type includes multiple specified encoding formats; and an encoding format of the file to be detected is determined from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

[0084] Optionally, in the embodiment, the storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of the storage medium would include one or more of a line of electrical connections, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] In an example embodiment of the present application, a computer program product is also provided, which includes a computer program, and the computer program, when executed by a processor, implements any of the above-described methods for detecting a file encoding.

[0086] Optionally, the computer program, when executed by the processor, can implement the following steps:

[0087] The file to be detected is received, a target byte of byte stream data corresponding to the file to be detected is obtained, and whether the file to be detected is of a standard encoding format type is determined according to the target byte; in a case where the file to be detected is of a non-standard encoding format type, byte stream segments are obtained by uniformly dividing byte stream data to which the file to be detected belongs, wherein the non-standard encoding format type includes multiple specified encoding formats; and an encoding format of the file to be detected is determined from an encoding format set corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

[0088] According to an embodiment of the present application, an electronic device is provided, which includes at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-described methods for detecting a file encoding.

[0089] Optionally, the electronic device can further include a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0090] Figure 4 This is a schematic block diagram of an electronic device 400 according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0091] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0092] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] The computing unit 401 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs various methods and processes described above, such as the method of detecting file encoding. For example, in some embodiments, the method of detecting file encoding can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded onto the RAM 403 and executed by the computing unit 401, one or more steps of the method of detecting file encoding described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the method of detecting file encoding by any other appropriate means, such as by means of firmware.

[0094] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0095] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0096] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include but are not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0098] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0099] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0100] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0102] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only illustrative, and for example, the division of units can be a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0103] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0104] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0105] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0106] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for detecting file encoding, characterized in that, include: Receive the file to be tested; Obtain the target byte of the byte stream data corresponding to the file to be detected, and determine whether the file to be detected is of a standard encoding format type based on the target byte; When the file to be detected is a non-standard encoding format type, the byte stream data to which the file to be detected belongs is evenly divided to obtain a set of byte stream segments, wherein the non-standard encoding format type includes multiple specified encoding formats; The encoding format of the file to be detected is determined from the set of encoding formats corresponding to the non-standard encoding format type based on the encoding format of each byte stream segment; The step of determining whether the file to be detected is a standard encoding format type based on the target byte includes: determining whether the target byte has a Byte Order Mark (BOM); if the target byte has the BOM, determining that the encoding format type of the file to be detected is a standard encoding format type; if the target byte does not have the BOM, determining that the encoding format type of the file to be detected is a non-standard encoding format type. Dividing the byte stream data of the file to be detected into uniform lengths includes: determining the total length of the byte stream data; determining a unit length based on the total length and a predetermined number, wherein the sum of the predetermined number of unit lengths is equal to the total length; and dividing the byte stream data into uniform lengths according to the unit lengths.

2. The method according to claim 1, characterized in that, The encoding format of the file to be detected is determined from the set of encoding formats corresponding to non-standard encoding format types based on the encoding format of each byte stream segment, including: Detect the encoding format of each byte stream segment; Determine the number of byte stream segments in the byte stream segment set that belong to the garbled format; Compare the number of byte stream segments with a preset threshold; Based on the comparison results, determine whether the file to be tested is in a garbled format.

3. The method according to claim 2, characterized in that, Determining whether the file to be tested is in garbled format based on the comparison results includes: If the number of byte stream segments exceeds the preset threshold, the file to be detected is determined to be in garbled format. If the number of byte stream segments is less than the preset threshold, determine the number of byte stream segments of other encoding formats in the byte stream segment set; The encoding format of the file to be detected is determined based on the number of byte stream segments of the other encoding formats, wherein the other encoding formats are encoding formats other than garbled formats.

4. The method according to claim 3, characterized in that, Determining the encoding format of the file to be detected based on the number of byte stream segments in the other encoding formats includes: The byte stream segment with the most frequent encoding format in the byte stream segment set is determined, and the most frequent encoding format is used as the encoding format of the file to be detected.

5. The method according to claim 4, characterized in that, Determining the byte stream segment with the most abundant encoding format in the byte stream segment set includes: If the number of byte stream segments with garbled format is the largest in the byte stream segment set and the number of byte stream segments with garbled format does not exceed the preset threshold, then determine the number of byte stream segments with other encoding formats in the byte stream segment set. Identify the byte stream segment with the most encoding format (excluding garbled formats) in the byte stream segment set.

6. A method for detecting file encoding, characterized in that, include: The target device receives the document to be tested; Determining the encoding type of the file to be tested includes: obtaining the target byte of the byte stream data corresponding to the file to be tested; determining whether the target byte has a byte order mark (BOM); if the target byte has the BOM, determining that the encoding format type of the file to be tested is a canonical encoding format type; if the target byte does not have the BOM, determining that the encoding format type of the file to be tested is a non-canonical encoding format type. The target detection method corresponding to the encoding type is determined from the detection methods pre-stored in the target device. The detection method is used to detect the encoding format of the file to be detected. The detection methods pre-stored in the target device include: a first detection method corresponding to a standard encoding format type and a second detection method corresponding to a non-standard encoding format type. The target detection method is used to determine the encoding format of the file to be detected; wherein, when the file to be detected is a non-standard encoding format type, the byte stream data to which the file to be detected belongs is evenly divided in the following manner to obtain a set of byte stream segments, and the encoding format of the file to be detected is determined from the set of encoding formats corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment: determining the total length of the byte stream data; determining the unit length according to the total length and a predetermined number, wherein the sum of the predetermined number of unit lengths is equal to the total length; and evenly dividing the byte stream data according to the unit length.

7. An apparatus for detecting file encoding, characterized in that, include: The receiving module is used to receive the file to be tested; The judgment module is used to obtain the target byte of the byte stream data corresponding to the file to be detected, and determine whether the file to be detected is a standard encoding format type based on the target byte; wherein the judgment method includes: determining whether the target byte has a byte order mark (BOM); if the target byte has the BOM, determining that the encoding format type of the file to be detected is a standard encoding format type; if the target byte does not have the BOM, determining that the encoding format type of the file to be detected is a non-standard encoding format type; The segmentation module is used to, when the file to be detected is a non-standard encoding format type, evenly divide the byte stream data to which the file to be detected belongs to obtain a set of byte stream segments, wherein the non-standard encoding format type includes multiple specified encoding formats; the segmentation method includes: determining the total length of the byte stream data; determining the unit length based on the total length and a predetermined number, wherein the sum of the predetermined number of unit lengths is equal to the total length; and evenly dividing the byte stream data according to the unit length; The determination module is used to determine the encoding format of the file to be detected from the set of encoding formats corresponding to the non-standard encoding format type according to the encoding format of each byte stream segment.

8. A non-volatile storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the method for detecting file encoding as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for detecting file encoding as described in any one of claims 1 to 6.

Images