File code detection method and device, equipment, storage medium and program product

Abstract

The invention provides a file coding detection method and device, equipment, a storage medium and a program product, and relates to the technical field of coding and decoding. The file code detection method comprises the steps that prior information of a target file is acquired, and the prior information is associated with an application scene of the target file; based on a plurality of preset coding formats, obtaining a plurality of candidate detection codes corresponding to the prior information, the candidate detection codes being in one-to-one correspondence with the preset coding formats; and respectively acquiring coding confidence coefficients of the plurality of candidate detection codes relative to the target file, and taking a preset coding format corresponding to the candidate detection code of which the coding confidence coefficient is highest or higher than a preset threshold value as a target coding format of the target file.

Description

Technical Field

[0001] This specification relates to the field of encoding and decoding technology, and in particular to a file encoding detection method, apparatus, device, storage medium, and program product. Background Technology

[0002] In daily work and application usage, there are often scenarios involving file transfer. When the recipient needs to view the specific content of a file provided by the sender, they need to first obtain the file's encoding format and then open the file according to that encoding format to view it.

[0003] Given the differences in default encoding formats between different operating systems, if the sender and receiver use different operating systems, or if the sender uses a special encoding format for the file and does not declare the encoding format when providing the file, the receiver may encounter garbled characters or other abnormalities when opening the file directly due to encoding format differences, making it impossible to correctly obtain the information in the file.

[0004] In view of this, embodiments of this specification provide a file encoding detection method, apparatus, device, computer-readable storage medium, and computer program product, which can accurately and efficiently identify the encoding format of files with unknown encoding formats. Summary of the Invention

[0005] One or more embodiments of this specification provide a file encoding detection method, the method comprising: obtaining prior information of a target file, the prior information being associated with the application scenario of the target file; obtaining multiple candidate detection codes corresponding to the prior information based on multiple preset encoding formats, the candidate detection codes corresponding one-to-one with the preset encoding formats; obtaining the encoding confidence of the multiple candidate detection codes relative to the target file, and using the preset encoding format corresponding to the candidate detection code with the highest encoding confidence or higher than a preset threshold as the target encoding format of the target file.

[0006] In some embodiments, the prior information includes at least one feature string, which is determined according to the application scenario of the target file and is contained in the plaintext content corresponding to the target file.

[0007] In some embodiments, obtaining multiple candidate detection codes corresponding to prior information based on multiple preset encoding formats includes: encoding each feature string contained in the prior information according to multiple preset encoding formats to obtain multiple encoded string sequences, and using each encoded string sequence as a candidate detection code corresponding to the prior information; wherein, the encoded string sequence is formed by concatenating the encoded strings corresponding to each feature string.

[0008] In some embodiments, obtaining the encoding confidence of multiple candidate detection codes relative to the target file, and using the preset encoding format corresponding to the candidate detection code with the highest encoding confidence or higher than a preset threshold as the target encoding format of the target file, includes: traversing the file encoding of the target file; when all the encoded strings contained in the candidate detection codes exist in the file encoding, using the encoding format corresponding to the candidate detection code set as the target encoding format; when only a portion of the encoded strings contained in the candidate detection codes exist in the file encoding, determining the encoding confidence based on the proportion of each encoded string contained in the candidate detection code set existing in the file encoding.

[0009] In some embodiments, the file encoding is obtained directly from the target file; or the file encoding is obtained by decoding the target file according to its file type using the format decoding method corresponding to the file type.

[0010] In some embodiments, prior information is located at a preset position in the target file; obtaining the encoding confidence of multiple candidate detection codes relative to the target file, and using the preset encoding format corresponding to the candidate detection code with the highest encoding confidence or higher than a preset threshold as the target encoding format of the target file, includes: determining the file code to be inspected based on prior information; traversing the file codes to be inspected; when all the encoding strings contained in the candidate detection codes exist in the file codes to be inspected, using the encoding format corresponding to the candidate detection codes as the target encoding format; when only a portion of the encoding strings contained in the candidate detection codes exist in the file codes to be inspected, determining the encoding confidence based on the proportion of each encoding string contained in the candidate detection code set that exists in the file codes.

[0011] In some embodiments, determining the file code to be inspected for a target file based on prior information includes: determining the length of the file code to be inspected based on the length of each feature string contained in the prior information, and determining the file code to be inspected for the target file.

[0012] In some embodiments, determining the file code length to be checked of the target file based on the length of each feature string contained in the prior information, and determining the file code to be checked of the target file, includes: obtaining the length of each encoding string sequence corresponding to each preset encoding format based on each feature string contained in the prior information; determining the longest length of each encoding string sequence as the file code length to be checked, and taking the file code of the target file up to the file code length from the preset position as the file code to be checked.

[0013] In some embodiments, the application scenario includes collecting questionnaire survey content; the target document includes the questionnaire.

[0014] In some embodiments, the file encoding detection method further includes: decoding the target file based on the target encoding format to obtain the plaintext content corresponding to the target file; verifying the plaintext content based on prior information, and determining the target encoding format as the verification encoding format of the target file when all feature strings contained in the prior information are contained in the plaintext content.

[0015] One or more embodiments of this specification also provide a file encoding detection device, the device comprising: an acquisition unit for acquiring prior information of a target file, the prior information being associated with the application scenario of the target file; an encoding unit for acquiring multiple candidate detection codes corresponding to the prior information based on multiple preset encoding formats, the candidate detection codes corresponding one-to-one with the preset encoding formats; and a detection unit for acquiring the encoding confidence of the multiple candidate detection codes relative to the target file, and using the preset encoding format corresponding to the candidate detection code with the highest encoding confidence or higher than a preset threshold as the target encoding format of the target file.

[0016] One or more embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the file encoding detection method described in some embodiments of this specification.

[0017] One or more embodiments of this specification also provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, can implement the file encoding detection method described in some embodiments of this specification.

[0018] One or more embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement the file encoding detection method described in some embodiments of this specification.

[0019] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: obtaining candidate detection codes corresponding to the prior information based on the prior information in the target file and the preset common encoding formats, and then using the candidate detection codes to perform retrieval and matching in the target file, which can accurately identify and obtain the encoding format corresponding to the target file, avoiding garbled characters due to incorrect encoding formats when reading the target file; when performing retrieval and matching in the target file, determining the range of the file's code to be detected based on the length and position of the prior information, improving the efficiency of identifying and obtaining the encoding format corresponding to the target file, and reducing the workload and time required for file encoding detection; further plaintext verification of the obtained target detection codes to ensure the accuracy of the identification of the encoding format corresponding to the target file. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may occur can be any one or a combination of the above, or any other possible beneficial effects. Attached Figure Description

[0020] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0021] Figure 1 This is a schematic diagram illustrating the application environment of a file encoding detection method according to some embodiments of this specification.

[0022] Figure 2 This is an exemplary flowchart of a file encoding detection method according to some embodiments of this specification.

[0023] Figure 3 This is a schematic diagram of a video frame of an original video stream according to some embodiments of this specification.

[0024] Figure 4 This is an exemplary flowchart illustrating an acquisition of a target encoding format according to some embodiments of this specification.

[0025] Figure 5 This is an exemplary flowchart illustrating another method for obtaining a target encoding format, according to some embodiments of this specification.

[0026] Figure 6 This is an exemplary flowchart illustrating an example of obtaining the code to be checked from a file, according to some embodiments of this specification.

[0027] Figure 7 This is an exemplary flowchart of a file encoding detection method according to some embodiments of this specification.

[0028] Figure 8This is an exemplary structural diagram of a document encoding detection device according to some embodiments of this specification.

[0029] Figure 9 This is a schematic diagram of the hardware architecture of a computer device according to some embodiments of this specification. Detailed Implementation

[0030] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0031] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other words can achieve the same purpose, they may be replaced by other expressions.

[0032] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0033] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0034] In daily work and application usage, file transfer scenarios are frequent. When the recipient needs to view the specific content of a file provided by the sender, they must first obtain the file's encoding format and then open the file according to that encoding format. In some embodiments, an encoding format is a rule that maps a character set to a sequence of bytes, determining how content information is stored as binary data so that a computer can understand and process it. In some embodiments, different encoding formats can be applied to different languages ​​and application scenarios, and the same content information can also be encoded using different encoding formats. In some embodiments, common encoding formats may include ASCII (American Standard Code for Information Interchange), ISO-8859 series (a series of 8-bit character set standards jointly developed by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), UTF (Unicode Transformation Format)-8 / 16 / 32, GBK (Chinese Internal Code Specification), etc. ASCII encoding allows 7 bits to represent 128 characters, including English letters, numbers, and some control characters. It is suitable for English text but does not support characters from other languages. ISO-8859-1 (Latin-1) encoding supports Western European languages, uses 8 bits, and can represent 256 characters. UTF-8 is a variable-length encoding format that allows 1 to 4 bytes to represent a Unicode character and is backward compatible with ASCII encoding, capable of representing almost all characters in the world. GBK encoding supports encoding and decoding of 20,902 Chinese characters.

[0035] Considering the differences in the default encoding formats of different operating systems, if the operating systems used by the sender and the receiver are different, or the sender uses a special encoding format for the file and does not declare the encoding format of the file when providing it, it is easy to cause abnormal situations such as garbled characters when the receiver directly opens the file due to encoding format differences, and the information in the file cannot be correctly obtained. Especially in some scenarios where files transmitted from multiple different information sources need to be uniformly collected and reviewed, for example, in a scenario of publicly soliciting questionnaire questions, the reviewers need to review the solicitation content provided by different solicitation sources. At this time, if there is a file garble caused by an unclear encoding format, it will seriously affect the review progress and work efficiency of the reviewers. In some embodiments, garbled characters are a situation of deviation in the file content after decoding the file using a mismatched encoding format. For example, the file contains the Chinese characters "为人" encoded in the GBK encoding format. If the UTF-8 format, which does not match the encoding format of the file, is used to open the file, the characters "为人" will be incorrectly displayed as the three characters "涓轰汉" with both meaning and content deviation, resulting in a garbled display situation.

[0036] In view of this, the embodiments of the present specification provide a technical solution for file encoding detection, which can accurately and efficiently identify the encoding format of a file with an unknown encoding format based on the prior information of the target file.

[0037] Figure 1 It is a schematic diagram of the application environment of a file encoding detection method shown according to some embodiments of the present specification. In some embodiments, as Figure 1 shown, the application environment 100 may include a computer device 110, a network 120, and a client 130, where the computer device 110 can be connected to the client 130 through the network 120. In some embodiments, the computer device 110 can perform file encoding detection on the target file provided by the client 130 to determine the target encoding format of the target file.

[0038] In some embodiments, the computer device 110 may be a high-performance computer device used to perform file encoding detection on the target file provided and transmitted by the client 130 to determine the target encoding format of the target file. In some embodiments, the computer device 110 may decode the target file based on the target encoding format and provide the decoded plaintext content to other clients 130 for viewing. In some embodiments, the computer device 110 may also provide the target file and the target encoding format together to other clients 130 for viewing. In some embodiments, the computer device 110 may be a single computer device or a computing cluster composed of multiple computer devices, thereby providing powerful computing power and efficient response for the target encoding format detection of the target file. In some embodiments, the computer device 110 may be a server, which may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Distribution Network), and big data and artificial intelligence platforms.

[0039] In some embodiments, network 120 can be any form of wired or wireless network, or any combination thereof. By way of example only, network 120 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, internal networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, etc. Network 120 can have multiple access points, through which computer devices 110 and clients 130 can access network 120.

[0040] In some embodiments, client 130 may include, but is not limited to, desktop computers, smartphones, laptops, VR (Virtual Reality) devices, and tablets. Users can browse, edit, and transfer target files on client 130, or view the plaintext content of target files provided by other clients 130; this is not limited to any particular device. In some embodiments, client 130 can receive and display the plaintext content of target files provided by computer device 110. In some embodiments, client 130 can accept target files and their corresponding target encoding formats provided by computer device 110, decode the target files based on the target encoding formats to obtain the plaintext content of the target files, and display it; this is not limited to any particular device.

[0041] It should be noted that, Figure 1 The illustrated application environment of the file encoding detection method shown is merely an example. The illustrated application environment described in the embodiments of this specification is intended to more clearly illustrate the technical solutions of the embodiments of this specification and does not constitute a limitation on the technical solutions provided in the embodiments of this specification. For example, Figure 1 The number of computer devices 110 and clients 130 in this specification is merely illustrative and is not intended to limit the scope of patent protection. Depending on the actual situation, any number of computer devices 110 and clients 130 may be used. For example, in some peer-to-peer file transfer applications, two or more clients 130 can directly connect and communicate via network 120, and data transfer does not need to go through computer devices 110. As those skilled in the art will understand, with the development of file transfer and the emergence of new business scenarios, the technical solutions provided in the embodiments of this specification are also applicable to similar technical problems.

[0042] The technical solutions described below will be illustrated through some embodiments, using computer device 110 or client 130 as the implementing entity. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0043] Figure 2 This is an exemplary flowchart illustrating a file encoding detection method according to some embodiments of this specification. In some embodiments, Figure 2 The process 200 shown can be executed by a terminal device, for example, by a terminal device such as... Figure 1 The client 130 shown executes; or, Figure 2 The process 200 shown can also be executed by other processing devices, for example, by... Figure 1 The computer device 110 shown performs the operation; or, Figure 2 The process 200 shown can be executed jointly by a terminal device and other processing devices, for example, it can be executed by... Figure 1 The client 130 and computer device 110 shown are executed together. In some embodiments, process 200 may be implemented by a file encoding detection device 800 deployed on a processing device and / or terminal device.

[0044] In some embodiments, such as Figure 2 As shown, process 200 may include the following steps.

[0045] Step 210: Obtain prior information about the target file. In some embodiments, step 210 can be implemented by the acquisition unit 810.

[0046] In some embodiments, prior information may be partial information contained in the target file that has been obtained before file encoding detection is performed on the target file. It may be content information contained in the target file that the user can obtain without knowing the correct encoding method of the target file. In some embodiments, prior information is associated with the application scenario of the target file. For example, if the target file is a template file provided to other users for content filling, then the template content in the template file can serve as the prior information of the target file. For instance, when the target file is a questionnaire template file or an answer template file, the prior information of the target file may be the template content related to the questionnaire or answer in the template file. In some embodiments, prior information may include various forms of content information such as text, images, and audio present in the target file, without limitation.

[0047] In some embodiments, prior information may include at least one feature string, which can be determined based on the application scenario of the target file and is contained in the plaintext content corresponding to the target file. In some embodiments, the feature string is associated with the application scenario of the target file and may be text or content that is necessarily contained in the target file. For example, when the target file is a questionnaire template file, the feature string it contains may be "question," "topic," "answer," etc. Figure 3 This is a schematic diagram illustrating the plaintext content of a target file containing prior information, according to some embodiments of this specification. In some embodiments, such as Figure 3 As shown, the target file 300 can be a question template file presented in tabular form. The table header includes "Question Content (suggested to be within 40 characters)," "Correct Answer (suggested to be within 30 characters)," "Answer Option 1," "Answer Option 2," etc. This header content is built into the template and is fixed; therefore, this header content, or at least a portion of it, can be used as prior information for the target file. In some embodiments, such as... Figure 3 The prior information of the target file 300 shown may include characteristic strings such as "question content (suggested to be within 40 characters)", "correct answer (suggested to be within 30 characters)", "answer option 1", and "answer option 2"; in other embodiments, such as Figure 3 The prior information of the target file 300 shown may include characteristic strings such as "question content", "correct answer", and "answer options", which are not limited here. In some embodiments, the prior information may also be content that is retained in the target file, such as in... Figure 3In the target file 300 shown, the prior information may also include relatively fixed content such as "The above are official example questions and are not included in the question statistics". Those skilled in the art can select appropriate plaintext content in the target file as the feature string included in the prior information according to actual needs, and no limitation is made here.

[0048] Step 220: Based on multiple preset encoding formats, obtain multiple candidate detection codes corresponding to the prior information. In some embodiments, the preset encoding formats can be pre-set according to the frequency of use of the encoding method in actual applications, mainly selecting commonly used encoding formats as preset encoding formats. In some embodiments, the candidate detection code is an encoded string obtained by encoding the prior information using the preset encoding format. In some embodiments, the candidate detection code corresponds one-to-one with the preset encoding format, that is, using one preset encoding format to encode the prior information to obtain a corresponding candidate detection code. In some embodiments, the multiple preset encoding formats may include ASCII encoding format, ISO-8859 series encoding format, UTF-8 / 16 / 32 encoding format, GBK encoding format, etc., and any combination of the above encoding formats, which are not limited here. In some embodiments, step 220 can be implemented by encoding unit 820.

[0049] In some embodiments, the prior information may include at least one feature string. Each feature string in the prior information may be encoded based on multiple preset encoding formats to obtain multiple encoded string sequences, and each encoded string sequence may be used as a candidate detection code corresponding to the prior information. In some embodiments, the encoded string sequence may be formed by concatenating the encoded strings corresponding to the feature strings in the prior information. In some embodiments, the prior information includes, for example,... Figure 3Taking the two feature strings "Question Content" and "Correct Answer" as an example, and assuming the preset encoding methods include UTF-8 and GBK encoding formats, the feature string "Question Content" can be represented as "E9A298E79BAEE58685E5AEB9" in UTF-8 encoding and "CCE2C4BFC4DAC8DD" in GBK encoding. Similarly, the feature string "Correct Answer" can be represented as "E6ADA3E7A1A" in UTF-8 encoding. The encoded string "EE7AD94E6A188" in GBK encoding format can be represented as "D5FDC8B7B4F0B0B8"; correspondingly, the encoded string sequence of prior information in UTF-8 encoding format can be represented as "E9A298E79BAEE58685E5AEB9E6ADA3E7A1AEE7AD94E6A188", while the encoded string sequence of prior information in GBK encoding format can be represented as "CCE2C4BFC4DAC8DDD5FDC8B7B4F0B0B8".

[0050] Step 230: Obtain the encoding confidence scores of multiple candidate detection codes relative to the target file, and use the preset encoding format corresponding to the candidate detection code with the highest encoding confidence score or higher than a preset threshold as the target encoding format of the target file. In some embodiments, step 230 can be implemented by the detection unit 830.

[0051] In some embodiments, the encoding confidence of a candidate detection code relative to the encoding of the target file can be used to characterize the likelihood that the preset encoding format corresponding to the candidate detection code is the target encoding format of the target file. The higher the encoding confidence, the more likely the preset encoding format corresponding to the candidate detection code is to be the target encoding format of the target file. In some embodiments, candidate detection codes corresponding to prior information are obtained based on prior information in the target file and preset common encoding formats. These candidate detection codes can then be used to perform retrieval and matching in the target file, accurately identifying and obtaining the encoding format corresponding to the target file, thus avoiding garbled characters due to incorrect encoding formats when reading the target file. The following will further explain and elaborate on the specific methods for obtaining encoding confidence and target encoding formats in conjunction with embodiments.

[0052] Figure 4 This is an exemplary flowchart illustrating an acquisition of a target encoding format according to some embodiments of this specification. In some embodiments, such as Figure 4 The illustrated process 400 can be implemented based on the detection unit 830. In some embodiments, such as Figure 4 As shown, process 400 may include the following steps.

[0053] Step 410: Traverse the file encoding of the target file. In some embodiments, the file encoding is a way to convert content information such as characters and images in the target file into binary data that can be recognized and processed by a computer. The same content in the target file will form different file encodings under different encoding formats. For example, the Chinese characters "你好" can be represented as the file encoding "0xE4%BD%A0%E5%A5%BD" in the UTF-8 encoding format; while in the GBK encoding format, its corresponding file encoding can be represented as "0xC4%E3%BA%C3". In some embodiments, the file encoding can be directly obtained based on the target file. For example, when the target file is a file in CSV (Comma-Separated Values) format, since it is in plain text format, any text editor can be used to directly open and read the file encoding. In some embodiments, the file encoding can also be obtained by performing format decoding on the target file according to the file type of the target file through the format decoding method corresponding to the file type. For example, when the target file is an Excel file, since the Excel file undergoes a secondary encoding related to the file format after encoding the plain text content, in actual operation, for target files in various file formats that have undergone secondary or multiple encodings, including the Excel format, format decoding needs to be performed first and then the file encoding is obtained. This is not limited here.

[0054] In some embodiments, the purpose of traversing the file encoding of the target file is to retrieve whether each encoding string contained in the candidate detection encoding exists in the file encoding, and determine the encoding confidence according to the retrieval and matching results. In some embodiments, methods such as linear search algorithm, binary search algorithm, KMP (Knuth-Morris-Pratt) algorithm, etc. can be selected to implement the traversal of the candidate detection encoding in the file encoding. This is not limited here. In some embodiments, considering that the feature strings contained in the prior information may be distributed at different positions in the target file, the entire file encoding of the target file can be traversed to achieve full coverage of the feature strings in the prior information, and avoid the situation where the file encoding corresponding to some feature strings in the prior information is not traversed due to too narrow traversal range.

[0055] Step 420: When all the encoded strings contained in the candidate detection codes exist in the file encoding, the encoding format corresponding to the candidate detection code set is taken as the target encoding format. In some embodiments, "all the encoded strings contained in the candidate detection codes exist in the file encoding" can mean that the encoded strings exist continuously in the file encoding, or that the encoded strings exist scattered in different parts of the file encoding. In some embodiments, when all the encoded strings contained in the candidate detection codes exist in the file encoding, it indicates that the encoded content of each feature string in the prior information under the current preset encoding format exists in the file encoding. At this time, it can be considered that the target file adopts the same encoding format as the current preset encoding format, and the encoding format corresponding to the candidate detection code set can be directly taken as the target encoding format of the target file.

[0056] Step 430: When each encoded string contained in the candidate detection encoding exists only partially in the file encoding, determine the encoding confidence based on the proportion of each encoded string contained in the candidate detection encoding set that exists in the file encoding.

[0057] In some embodiments, when all candidate detection codes are only partially present in the file encoding, it can be considered that the currently selected preset encoding format does not belong to the target encoding format of the target file. The currently selected preset encoding format can be excluded from the target encoding format of the target file, and the manual intervention process can be initiated. Other encoding formats can be selected to repeat the above steps to detect whether other encoding formats belong to the target encoding format of the target file.

[0058] In some embodiments, when each encoded string contained in the candidate detection code exists only partially in the file code, it indicates that there is a difference between the result of encoding the prior information by the preset encoding format corresponding to the candidate detection code and the file code. This may be because the preset encoding format corresponding to the candidate detection code does not conform to the target file, or it may be that the specific content of the prior information in the target file has changed. For example, if the target file is a questionnaire template provided to users for completion, the prior information may be the content of the fixed header section in the questionnaire template. However, if the user makes some adjustments and / or deletions to the content of the header section when providing feedback on the target file (e.g., changing the content of the header section), the prior information may be different. Figure 3 If the "Answer Option 1" shown is changed to "Answer Option a"), it may result in the incomplete existence of all encoded strings contained in the candidate detection code within the file encoding. In such cases, the preset encoding format corresponding to the candidate detection code can be evaluated based on the encoding confidence level.

[0059] In some embodiments, the encoding confidence score can be determined based on the proportion of each encoded string in the candidate detection encoding set that exists in the file encoding. The higher the proportion of each encoded string in the candidate detection encoding set that exists in the file encoding, the more feature strings exist in the prior information under the current preset encoding format. In some embodiments, considering that different encoding formats may have the same encoded content for some strings, such as UTF-16 encoding format and UTF-8 encoding format having the same encoded content for ASCII strings, the preset threshold for encoding confidence score can be set to a relatively high value. For example, the preset threshold for encoding confidence score can be set to 0.85 or 0.95, which requires that the proportion of each encoded string in the candidate detection encoding set that exists in the file encoding needs to reach 85% or 95% or more. Those skilled in the art can also set the preset threshold for encoding confidence score according to actual needs, which is not limited here. In some embodiments, by setting an encoding confidence level, candidate detection codes can be screened according to a unified standard based on the level of encoding confidence. The comparison between the encoding confidence level and a preset threshold determines whether the candidate detection code belongs to the target encoding format of the target file. This can improve the efficiency and accuracy of obtaining the target encoding format in special scenarios such as changes in prior information and the specific content of the target file.

[0060] Based on the descriptions in the foregoing embodiments, when acquiring and determining the target encoding format, candidate detection codes can be traversed and searched throughout the entire file encoding of the target file. When the target file is large, traversing and searching all file encodings may consume significant time and computational resources. In some embodiments, the prior information of the target file may include not only the specific content of the target file but also the location of the prior information within the target file. Therefore, based on the preset location of the prior information in the target file, the file encodings to be traversed and searched can be further narrowed down, thereby improving the overall efficiency of file encoding detection and reducing computational resource consumption. Specifically, Figure 5 This is an exemplary flowchart illustrating another method for obtaining a target encoding format, according to some embodiments of this specification. In some embodiments, such as Figure 5 The illustrated process 500 can be applied to scenarios where prior information is located at a preset location in the target file and can be obtained, such as when the prior information includes... Figure 3 In the application scenarios shown, such as the "question content," "correct answer," and "answer options," since the feature strings are used as the header of a table, their encoded form is located at the very beginning of the file encoding portion of the target file—that is, the prior information is located at the very beginning of the target file, a predetermined position. In some embodiments, such as... Figure 5The illustrated process 500 can be implemented based on the detection unit 830. In some embodiments, such as Figure 5 As shown, process 500 may include the following steps.

[0061] Step 510: Based on prior information, determine the file code to be checked in the target file. In some embodiments, since the prior information is located at a preset position in the target file, a file code of a certain length near the preset position can be selected as the file code to be checked in the target file. In some embodiments, by searching and matching candidate detection codes in the file code to be checked, the consumption of computing resources and the time spent on traversal retrieval will be reduced compared to performing a full-text traversal retrieval of the file code of the target file. It is understood that since the length of the file code to be checked is usually smaller than the length of the file code of the entire target file, the time required to traverse the file code to be checked is also smaller than the time required to traverse the file code of the target file. Especially in scenarios where the target file is large, using the file code to be checked instead of a full-text traversal of the target file can significantly reduce the overall time spent on file code detection and improve the efficiency of file code detection.

[0062] In some embodiments, when determining the file code to be checked, the length of the code to be checked can be determined based on the length of each feature string contained in the prior information, and the file code to be checked in the target file can be determined. In some embodiments, when determining the length of the code to be checked, the file code to be checked in the target file can be determined according to the length of the code to be checked and the preset position of the prior information in the target file. For example, when the prior information is located at the beginning or end of the target file, bytes of the code to be checked can be selected from the beginning or end of the file code backwards or forwards as the file code to be checked in the target file. The specific process for determining the file code to be checked will be further explained below with reference to embodiments.

[0063] Specifically, Figure 6 This is an exemplary flowchart illustrating an method for obtaining a file's code to be checked, according to some embodiments of this specification. In some embodiments, such as... Figure 6 As shown, process 600 may include the following steps.

[0064] Step 610: Based on the feature strings contained in the prior information, obtain the length of each encoded string sequence corresponding to each preset encoding format. In some embodiments, taking the feature string "Question Content" contained in the prior information and the preset encoding methods including UTF-8 encoding format, GBK encoding format, and ISO-8859-1 encoding format as an example, the encoded string of the feature string "Question Content" in UTF-8 encoding format can be represented as "E9A298E79BAEE58685E5AEB9", the encoded string in GBK encoding format can be represented as "CCE2C4BFC4DAC8DD", and the encoded string in ISO-8859-1 encoding format can be represented as "0x3F0x3F0x3F0x3F". It can be seen that the length of the encoded string of the feature string "Question Content" in UTF-8 encoding format is the longest. In some embodiments, by obtaining the length of each encoded string sequence in each preset encoding format, the length of the file to be checked encoding can be determined accordingly.

[0065] Step 620: Determine the longest length of each encoded string sequence as the file code to be checked, and take the file code of the target file up to the length of the code to be checked from the preset position as the file code to be checked. In some embodiments, taking the prior information containing the feature string "question content" and the preset encoding methods including UTF-8 encoding format, GBK encoding format, and ISO-8859-1 encoding format as an example, since the feature string "question content" has the longest encoded string in UTF-8 encoding format, in order to ensure that the file code to be checked can cover the encoded string sequence corresponding to each preset encoding format, the length of the longest encoded string in UTF-8 encoding format can be taken as the file code to be checked, and based on the position of the prior information in the target file, take the file code of the target file up to the length of the code to be checked from the preset position as the file code to be checked. In some embodiments, the preset position can be located at the beginning of the target file, the middle of the target file, or the end of the target file; correspondingly, the file code to be checked can be the file code of the length to be checked before the preset position, the file code of the length to be checked after the preset position, or the file code of the length to be checked within a certain range before and after the preset position, without limitation. In some embodiments, considering that in some specific application scenarios, the feature strings contained in the prior information may be scattered in different parts of the target file and do not appear continuously, when determining the length of the file code to be checked, it is not only necessary to consider the longest length of each encoded string sequence, but also to consider the interval length between each feature string on the basis of the longest length, that is, the length of the file code to be checked needs to be at least greater than the longest length of each encoded string sequence. In some embodiments, in scenarios where the feature strings included in the prior information appear consecutively in the target file, to ensure that the file to be checked encoding can completely cover the prior information and to avoid the situation where the delimiters or format switching characters between the feature strings occupy the encoded bytes, thus causing the file to be checked encoding to not completely cover the prior information, the length of the file to be checked encoding can also be set to be slightly larger than the longest length of each encoded string sequence. In some embodiments, when the prior information includes, for example, ... Figure 3 In the application scenarios shown for the feature strings "Question Content", "Correct Answer", "Answer Option 1", "Answer Option 2", "Answer Option 3", and "Answer Option 4", since the feature strings are used as the header of the table, their encoded value is located at the very beginning of the file encoding portion of the target file. Furthermore, these feature strings, which serve as prior information, are separated by other content, such as... Figure 3The content shown, such as "suggested within 30 characters" and "()", means that the first few bytes (e.g., 1024 bytes) of the target file can be used as the file to be checked to ensure that the file to be checked can completely cover the plaintext content corresponding to the prior information. No limit is imposed here.

[0066] Step 520: Traverse the file's codes to be checked. In some embodiments, the file's codes to be checked are traversed to search for matches to determine if each code string contained in the candidate detection codes exists. In some embodiments, step 520 can be implemented in a similar manner to step 410 in the foregoing embodiments, and is not limited thereto.

[0067] Step 530: When all the encoded strings contained in the candidate detection code exist in the file to be detected, the encoding format corresponding to the candidate detection code is taken as the target encoding format.

[0068] Step 540: When each encoded string contained in the candidate detection code exists only partially in the file code to be detected, determine the encoding confidence based on the proportion of each encoded string contained in the candidate detection code set that exists in the file code.

[0069] In some embodiments, the specific implementation of steps 530 and 540 can refer to the specific implementation of steps 420 and 430 in the foregoing embodiments, and will not be repeated here.

[0070] In some embodiments, the application scenario may include collecting questionnaire survey content; the target file includes the questionnaire. In some embodiments, the membership certification assessment of video platforms often requires a large number of assessment questions and / or questionnaire surveys, and the content of the assessment questions and / or questionnaire surveys needs to be constantly updated over time. In this case, questionnaire survey content can be collected from registered members and / or video creators of the video platform through questionnaire solicitation, or content can be collected from registered members and / or video creators through questionnaire surveys, and the collected questionnaires are reviewed by the staff of the video platform. Due to the differences in the operating systems or computer environments used by different registered members and / or video creators, it is difficult to open a large number of feedback questionnaires using the same encoding method, which easily leads to a large number of garbled characters in the files. In this scenario, the file encoding detection method provided in this specification can realize the automated encoding format detection of a large number of feedback questionnaires, and can accurately and efficiently detect the encoding format of target files with unknown encoding formats, avoiding garbled characters due to encoding format errors when reading target files.

[0071] In some embodiments, the target encoding format can be the encoding format in which each encoded string contained in the candidate detection encoding exists in the file encoding and / or the file to be detected encoding, or it can be the encoding format with the highest encoding confidence or higher than a preset threshold among the preset encoding formats. When the target encoding format is the encoding format with the highest encoding confidence or higher than the preset threshold among the preset encoding formats, there may be a situation where the target encoding format is inconsistent with the correct encoding format of the target file. For example, if the target file's correct encoding format is UTF-16, and the prior information includes multiple ASCII strings and one Chinese character string, the UTF-16 and UTF-8 encoding formats are highly similar in their representation of ASCII strings (the encoded content is the same, but the number of bytes occupied by the encoding differs; for example, the character "A" is represented as "0041" in UTF-16 and "41" in UTF-8). However, the UTF-16 and UTF-8 encoding formats differ in their representation of Chinese character strings. If the preset encoding format only includes UTF-8 and other encoding formats but not UTF-16, it's easy to mistakenly determine that UTF-8 is the target encoding format based on the prior information. However, decoding the target file using UTF-8 can still easily result in garbled characters. To avoid similar situations, a further verification process can be implemented for the obtained target encoding format. Specifically, Figure 7 This is an exemplary flowchart of a file encoding detection method according to some embodiments of this specification. In some embodiments, such as Figure 7 The illustrated process 700 can be implemented based on the verification unit 840. In some embodiments, such as Figure 7 As shown, process 700 may include the following steps.

[0072] Step 710: Decode the target file based on the target encoding format to obtain the plaintext content corresponding to the target file. It is important to note that the plaintext content corresponding to the target file here is not necessarily the actual plaintext content of the target file; it is only obtained by decoding the target file based on the target encoding format. It may be the same as the actual plaintext content of the target file, or it may differ. For example, if the target encoding format is the correct encoding format for the target file, the plaintext content obtained by decoding the target file based on the target encoding format will be consistent with the actual plaintext content of the target file; however, if the target encoding format is not the correct encoding format for the target file, the plaintext content obtained by decoding the target file based on the target encoding format may contain parts that are decoded correctly and parts that are garbled.

[0073] Step 720: Verify the plaintext content based on prior information. When all the feature strings contained in the prior information are contained in the plaintext content, determine the target encoding format as the verification encoding format of the target file.

[0074] In some embodiments, when the plaintext content obtained after decoding the target file using the target encoding format contains the feature strings from the prior information, for example, when the prior information includes such... Figure 3 In the application scenarios shown for "Question Content," "Correct Answer," and "Answer Options," if the plaintext content parsed using the target encoding format also contains the characteristic strings from the prior information such as "Question Content," "Correct Answer," and "Answer Options," then the target encoding format can be determined as the verification encoding format for the target file, thus determining the correct encoding format for the target file. However, if the plaintext content parsed using the target encoding format does not contain all the characteristic strings from the prior information such as "Question Content," "Correct Answer," and "Answer Options," it indicates that the target encoding format cannot completely decode the target file. The high encoding confidence may be due to the high similarity between the target encoding format and the correct encoding format of the target file in terms of encoding the prior information. In this case, the encoding format of the target file can be re-selected and set.

[0075] Figure 8 This is an exemplary structural diagram of a document encoding detection device according to some embodiments of this specification. In some embodiments, such as Figure 8 As shown, the file encoding detection device 800 may include an acquisition unit 810, an encoding unit 820, and a detection unit 830.

[0076] In some embodiments, the acquisition unit 810 can be used to acquire prior information of the target file, wherein the prior information is associated with the application scenario of the target file. In some embodiments, the encoding unit 820 can be used to acquire multiple candidate detection codes corresponding to the prior information based on multiple preset encoding formats, wherein the candidate detection codes correspond one-to-one with the preset encoding formats. In some embodiments, the detection unit 830 can be used to acquire the encoding confidence of multiple candidate detection codes relative to the target file, and use the preset encoding format corresponding to the candidate detection code with the highest encoding confidence or higher than a preset threshold as the target encoding format of the target file. In some embodiments, the detection unit 830 can also implement the processing of process 400 and / or process 500 provided in the foregoing embodiments, which is not limited here.

[0077] In some embodiments, such as Figure 8The file encoding detection device 800 shown may further include a verification unit 840. In some embodiments, the verification unit 840 may decode the target file based on the target encoding format to obtain the plaintext content corresponding to the target file, and verify the plaintext content based on prior information. When all feature strings contained in the prior information are contained in the plaintext content, the target encoding format is determined to be the verification encoding format of the target file.

[0078] For more information on each module, device, and system, please refer to [link / reference]. Figures 2 to 7 The relevant explanations will not be repeated here. It should be understood that... Figure 8 The apparatus and modules shown can be implemented in various ways. For example, in some embodiments, they can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods, apparatus, and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as in the memory of a disk, CD, or DVD-ROM. The apparatus and modules described herein can be implemented not only by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips or transistors, or programmable hardware devices such as field-programmable gate arrays or programmable logic devices, but also by software, for example, executed by various types of processors, or by a combination of the aforementioned hardware circuitry and software (e.g., firmware).

[0079] It should be noted that the above description of the device and modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principle of the device, can arbitrarily combine the various modules without departing from this principle to form sub-devices connected to other modules. Alternatively, some modules can be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0080] Figure 9This is a schematic diagram of the hardware architecture of a computer device 110 according to some embodiments of this specification. In some embodiments, the computer device 110 may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the file encoding detection method provided in any of the foregoing embodiments. In some embodiments, the computer device 110 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, robot, etc. In other embodiments, the computer device 110 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc.

[0081] In some embodiments, such as Figure 9 As shown, the computer device 110 includes, but is not limited to, a memory 111, a processor 112, and a network interface 113 that can communicate with each other via a system bus. The memory 111 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD memory, DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 111 may be an internal storage module of the computer device 110, such as the hard disk or RAM of the computer device 110. In other embodiments, the memory 111 may also be an external storage device of the computer device 110, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 111 may include both the internal storage module and the external storage device of the computer device 110. In this embodiment, the memory 111 is typically used to store the operating system and various application software installed on the computer device 110, such as the program code related to the file encoding detection method provided in any of the foregoing embodiments. Furthermore, the memory 111 can also be used to temporarily store various types of data that have been output or will be output.

[0082] In some embodiments, processor 112 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chip. Processor 112 is typically used to control the overall operation of computer device 110, such as performing control and processing related to data interaction or communication with computer device 110. In this embodiment, processor 112 is used to run program code stored in memory 111 or process data.

[0083] Network interface 113 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 110 and other computer devices. For example, network interface 113 is used to connect computer device 110 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 110 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, WiFi, or other wireless or wired networks.

[0084] It should be pointed out that, Figure 9 Only computer devices having components 111 to 113 are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented alternatively. In some embodiments, the relevant program code for the file encoding detection method stored in memory 111 may also be divided into one or more program modules and executed by one or more processors (such as processor 112) to implement the embodiments of this specification, without limitation herein.

[0085] One or more embodiments of this specification also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the file encoding detection method provided in the foregoing embodiments. In some embodiments, the computer-readable storage medium may include flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), a flash card, etc. Of course, the computer-readable storage medium may also include both internal storage units and external storage devices of a computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the file encoding detection method in this embodiment. Furthermore, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0086] One or more embodiments of this specification also provide a computer program product, including a computer program that, when executed by a processor, can implement the file encoding detection method provided in any of the foregoing embodiments. In some embodiments, the computer program product may only involve a computer program, which may be carried on a storage medium or a processing device. In other embodiments, the computer program product may also be a storage medium or a processing device containing the aforementioned computer program. The processing device may include one or more processors, and a storage medium.

[0087] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. A method for detecting file encoding, characterized in that, The method includes: Obtain prior information about the target file, wherein the prior information is associated with the application scenario of the target file; Based on multiple preset encoding formats, multiple candidate detection codes corresponding to the prior information are obtained, and the candidate detection codes correspond one-to-one with the preset encoding formats. The encoding confidence scores of the multiple candidate detection codes relative to the target file are obtained respectively, and the preset encoding format corresponding to the candidate detection code with the highest encoding confidence score or higher than a preset threshold is taken as the target encoding format of the target file.

2. The method according to claim 1, characterized in that, The prior information includes at least one feature string, which is determined based on the application scenario of the target file and is contained in the plaintext content corresponding to the target file.

3. The method according to claim 2, characterized in that, The process of obtaining multiple candidate detection codes corresponding to the prior information based on various preset encoding formats includes: Based on the aforementioned multiple preset encoding formats, each of the feature strings contained in the prior information is encoded to obtain multiple encoded string sequences, and each encoded string sequence is used as a candidate detection code corresponding to the prior information. The encoded string sequence is formed by concatenating the encoded strings corresponding to each of the feature strings.

4. The method according to claim 3, characterized in that, The step of obtaining the encoding confidence scores of the plurality of candidate detection codes relative to the target file, and using the preset encoding format corresponding to the candidate detection code with the highest encoding confidence score or higher than a preset threshold as the target encoding format of the target file, includes: Iterate through the file codes of the target file; When all the encoded strings contained in the candidate detection codes exist in the file encoding, the encoding format corresponding to the candidate detection code set is taken as the target encoding format; When each encoded string contained in the candidate detection encoding exists only partially in the file encoding, the encoding confidence is determined based on the proportion of each encoded string contained in the candidate detection encoding set that exists in the file encoding.

5. The method according to claim 4, characterized in that, The file encoding is obtained directly from the target file; or The file encoding is obtained by decoding the target file according to its file type using the format decoding method corresponding to that file type.

6. The method according to claim 3, characterized in that, The prior information is located at a preset position in the target file; The step of obtaining the encoding confidence scores of the plurality of candidate detection codes relative to the target file, and using the preset encoding format corresponding to the candidate detection code with the highest encoding confidence score or higher than a preset threshold as the target encoding format of the target file, includes: Based on the prior information, the file code to be inspected for the target file is determined; Iterate through the files to be checked for encoding; When all the encoded strings contained in the candidate detection code exist in the file to be detected code, the encoding format corresponding to the candidate detection code is taken as the target encoding format; When each of the coded strings contained in the candidate detection codes exists only partially in the file code to be detected, the coding confidence is determined based on the proportion of each of the coded strings contained in the candidate detection code set that exists in the file code.

7. The method according to claim 6, characterized in that, The step of determining the file code to be inspected for the target file based on prior information includes: Based on the length of each of the feature strings contained in the prior information, the length of the file code to be inspected is determined, and the file code to be inspected for the target file is determined.

8. The method according to claim 7, characterized in that, The step of determining the file inspection code length of the target file based on the length of each of the feature strings contained in the prior information, and determining the file inspection code of the target file, includes: Based on the feature strings contained in the prior information, the length of each encoded string sequence corresponding to each preset encoding format is obtained; The longest length of each encoded string sequence is determined as the length of the file code to be inspected, and the file code of the target file up to the specified length of the code to be inspected from the preset position is taken as the file code to be inspected.

9. The method according to claim 1, characterized in that, The application scenarios include collecting questionnaire survey content; The target document includes a questionnaire.

10. The method according to claim 1, characterized in that, The method further includes: The target file is decoded based on the target encoding format to obtain the plaintext content corresponding to the target file; The plaintext content is verified based on the prior information. When all the feature strings contained in the prior information are included in the plaintext content, the target encoding format is determined to be the verification encoding format of the target file.

11. A document encoding detection device, characterized in that, include: The acquisition unit is used to acquire prior information of the target file, wherein the prior information is associated with the application scenario of the target file; The encoding unit is used to obtain multiple candidate detection codes corresponding to the prior information based on multiple preset encoding formats, wherein the candidate detection codes correspond one-to-one with the preset encoding formats; The detection unit is used to obtain the encoding confidence of the plurality of candidate detection codes relative to the target file, and to take the preset encoding format corresponding to the candidate detection code with the highest encoding confidence or higher than a preset threshold as the target encoding format of the target file.

12. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 10.

Images