File processing method, device and system

Abstract

The present invention discloses a file processing method, device and system. The method is applied to a processing node of a distributed processing cluster, and the method comprises: obtaining a file to be split and a splitting rule corresponding to the file to be split, wherein the file to be split contains multiple fields; identifying multiple fields in the file to be split; matching the multiple fields with the splitting rule to determine the splitting fields in the multiple fields; splitting the file to be split based on the splitting fields to obtain multiple sub-files. The present invention solves the technical problem that the file processing method in the related art has a low splitting rate when splitting files.

Description

Technical Field

[0001] The present invention relates to the field of file processing, and in particular to a file processing method, device and system. Background Art

[0002] In the system architecture of the banking industry and other industries with headquarter and branch offices, there is a need to split the entire file into one or more sub-files. In the related art, the Linux operating system provides a split command to serially split files of equal size, but the splitting efficiency is low.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a file processing method, device and system to at least solve the technical problem of low efficiency in splitting files in the file processing method in the related art.

[0005] According to one aspect of an embodiment of the present invention, a file processing method is provided, which is applied to a processing node of a distributed processing cluster, wherein the method includes: obtaining a file to be split and a splitting rule corresponding to the file to be split, wherein the file to be split contains multiple fields; identifying multiple fields in the file to be split; matching the multiple fields with the splitting rule to determine the splitting fields among the multiple fields; and splitting the file to be split based on the splitting fields to obtain multiple sub-files.

[0006] Optionally, multiple fields are matched with a splitting rule to determine a splitting field among the multiple fields, including: generating a regular expression based on the splitting rule; matching the multiple fields with the regular expression; and determining a field that successfully matches the regular expression as a splitting field.

[0007] Optionally, before splitting the file to be split based on the splitting field to obtain multiple sub-files, the method also includes: obtaining data verification rules and data corresponding to the splitting fields in the file to be split; based on the data verification rules, verifying the data corresponding to the splitting fields; if the verification is successful, splitting the file to be split based on the splitting fields; if the verification fails, storing the data corresponding to the splitting fields in an error file.

[0008] Optionally, obtaining data corresponding to the split field in the to-be-split file includes: reading the data in the to-be-split file line by line; and determining the data corresponding to the split field based on a field delimiter corresponding to the split field.

[0009] Optionally, after splitting the file to be split based on the split field to obtain multiple sub-files, the method also includes: obtaining a first data volume of the multiple sub-files and a second data volume of the error file; obtaining a split ratio corresponding to the split field based on the first data volume and the second data volume; and determining a target split field based on the split ratio, wherein the target split field is used for the next splitting of the file to be split.

[0010] Optionally, when there are multiple split fields, the file to be split is split based on the multiple split fields, the split ratio corresponding to each split field is determined, and the split field corresponding to the highest split ratio is determined as the target split field.

[0011] Optionally, the splitting rules are obtained from a relational database or a distributed storage system.

[0012] According to another aspect of an embodiment of the present invention, a file processing device is also provided, which is deployed in a processing node of a distributed processing cluster, wherein the device includes: an acquisition module, used to acquire a file to be split, and a splitting rule corresponding to the file to be split, wherein the file to be split contains multiple fields; an identification module, used to identify multiple fields in the file to be split; a matching module, used to match multiple fields with the splitting rule, and determine the splitting field among the multiple fields; and a splitting module, used to split the file to be split based on the splitting field to obtain multiple sub-files.

[0013] According to another aspect of an embodiment of the present invention, a file processing system is also provided, including: a database, used to store splitting rules corresponding to files to be split, wherein the files to be split contain multiple fields; a distributed processing cluster, connected to the database, including processing nodes, the distributed processing cluster is used to store the files to be split, the processing nodes are used to identify multiple fields in the files to be split, match the multiple fields with the splitting rules, determine the splitting fields in the multiple fields, and split the files to be split based on the splitting fields to obtain multiple sub-files.

[0014] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned file processing method.

[0015] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the above-mentioned file processing method is executed when the program is run.

[0016] In an embodiment of the present invention, after obtaining the file to be split and the splitting rules corresponding to the file to be split, the multiple fields of the file to be split that are automatically identified can be matched with the splitting rules, thereby obtaining the splitting fields of the file to be split, and the split files are split according to the splitting fields, thereby achieving the purpose of automatic splitting of large batches of files. It is easy to notice that a large number of files can be split in parallel through a distributed processing cluster, and the splitting fields can be automatically identified, which reduces the workload of manual operation and avoids the problems of misoperation and splitting errors caused by manual operation, thereby achieving the technical effect of improving splitting efficiency and splitting accuracy and enhancing user experience, thereby solving the technical problem of low splitting of files in the file processing method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a file processing method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of an optional file splitting system architecture according to an embodiment of the present invention;

[0020] Figure 3 is a flowchart of an optional file splitting process according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a file processing device according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a file processing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The above-mentioned solution of using the split command to perform serial splitting of files into equal-sized files can only be performed on a single machine, and the splitting efficiency is limited by the CPU and memory limitations of a single server; the splitting function only splits the file into equal-sized files or a fixed number of lines, which makes it difficult to filter problematic data; if a failure occurs when splitting a large file, it is necessary to start over; in addition, this method cannot specify personalized differential rules and has poor flexibility.

[0026] The solution provided by the present invention aims to expand and improve the functions of the existing technology in view of the defects and deficiencies. The specific implementation scheme is as follows:

[0027] Example 1

[0028] According to an embodiment of the present invention, a file processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Optionally, the method may be applied to a processing node in a distributed processing cluster.

[0030] The above-mentioned distributed processing cluster can be a Hadoop cluster. MapReduce in the Hadoop cluster is a parallel computing model, framework and platform for large-scale data processing. The processing node can be a node in the Hadoop cluster. Each node can be deployed on a server, or multiple nodes can be deployed on the same server.

[0031] In an optional embodiment, a distributed splitting method can be used to achieve the purpose of splitting a large number of files at the same time, thereby greatly improving the splitting efficiency.

[0032] Figure 1is a flowchart of a file processing method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0033] Step S102, obtaining a file to be split and a splitting rule corresponding to the file to be split, wherein the file to be split includes multiple fields.

[0034] The files to be split in the above steps may be files that need to be split by each node in the distributed processing cluster. Each node may split different files at the same time, and may split the same file at the same time.

[0035] The splitting rules in the above steps can be different splitting rules customized by the user for different files, and the splitting range can be expanded at any time by setting the splitting rules. In an optional embodiment, the user can write splitting rules based on several known splitting fields, such as institution number, customer number, customer account number, province code, etc.

[0036] Optionally, the splitting rules are obtained from a relational database or a distributed storage system.

[0037] The above-mentioned relational database can be Oracle, MySQL or DB2, etc., but is not limited thereto. The above-mentioned distributed storage system can be HBase, which is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. The powerful reading capability provided by HBase can meet the business scenario of frequently reading rule data (Key / Value) for associated splitting processing during the splitting process.

[0038] In an optional embodiment, a relational database can be used as a splitting rule configuration library, mainly used to store splitting rules, splitting method configuration, files to be split, etc. Splitting rule data (Key / Value) is processed according to certain specific rule files (containing rule data that other file splitting depends on) and stored in HBase.

[0039] Step S104, identifying multiple fields in the file to be split.

[0040] The fields in the above steps may be all fields contained in the file to be split, such as institution number, customer number, customer account number, province code, etc., but are not limited thereto.

[0041] Step S106, matching the multiple fields with the splitting rule to determine the splitting field among the multiple fields.

[0042] The split field in the above steps may refer to a field that matches the split rule, that is, the split field is the field based on which the user writes the split rule. It should be noted that the user often bases on multiple fields when writing the split rule, so the identified split fields are often multiple.

[0043] Step S108, splitting the file to be split based on the splitting field to obtain multiple sub-files.

[0044] In an optional embodiment, when a large number of files need to be split, a distributed splitting method can be used to process them simultaneously. For each processing node, the splitting rules can be read from the database according to the splitting task, and the files to be split that the processing node needs to split can be obtained. All fields in the files to be split are identified, and all splitting fields that can be used for splitting are automatically identified based on the splitting rules, and splitting is performed accordingly.

[0045] Through the above-mentioned embodiment of the present invention, after obtaining the file to be split and the splitting rules corresponding to the file to be split, the multiple fields of the file to be split that are automatically identified can be matched with the splitting rules, so as to obtain the splitting fields of the file to be split, and the split files are split according to the splitting fields, so as to achieve the purpose of automatic splitting of large batches of files. It is easy to notice that a large number of files can be split in parallel through a distributed processing cluster, and the splitting fields can be automatically identified, which reduces the workload of manual operation and avoids the problems of misoperation and splitting errors caused by manual operation, thereby achieving the technical effect of improving splitting efficiency and splitting accuracy and enhancing user experience, thereby solving the technical problem of low splitting of files in the file processing method in the related art.

[0046] Optionally, in the above embodiment of the present invention, multiple fields are matched with splitting rules to determine split fields among the multiple fields, including: generating a regular expression based on the splitting rules; matching multiple fields with the regular expression; and determining that the field that successfully matches the regular expression is the split field.

[0047] A regular expression is a logical formula for string operations. It uses predefined specific characters and combinations of specific characters to form a regular string. The regular string can be used to implement logic such as string query and filtering.

[0048] In an optional embodiment, in order to facilitate the computer to automatically identify split fields, the user can directly write corresponding regular expressions based on several known types of split fields, so that each processing node can directly match fields through regular expressions when performing file splitting operations, and identify the fields that match the regular expression as split fields of the data to be split.

[0049] Optionally, in the above embodiment of the present invention, before splitting the file to be split based on the split field to obtain multiple sub-files, the method also includes: obtaining data verification rules, and data corresponding to the split field in the file to be split; based on the data verification rules, verifying the data corresponding to the split field; if the verification is successful, splitting the file to be split based on the split field; if the verification fails, storing the data corresponding to the split field in an error file.

[0050] The data verification rules in the above steps may be rules for verifying data fields customized by users for different files, including but not limited to data length, data type, numerical accuracy, etc. The data verification rules may also be stored in the above database.

[0051] The error file in the above steps may be a file used to store problem data (including error data, abnormal data, verification failure data, etc.).

[0052] In an optional embodiment, in order to avoid the situation where the file cannot be split due to data quality problems, the data in the file to be split can be read line by line and split according to the split fields, and then the data of the split field is checked according to the data verification rules to see if there are data quality problems. If the verification is successful, it means that there are no data quality problems with the data of the split field, and the split operation can continue; if the verification fails, it means that there are data quality problems with the data of the split field, and it can be directly written into the ERROR file. Through data verification, the problem data can be filtered and archived, which is convenient for subsequent investigation of the reasons for the failure to split and avoids the problem of data being unable to load or data inconsistency during use.

[0053] Optionally, in the above embodiment of the present invention, obtaining data corresponding to the split field in the file to be split includes: reading the data in the file to be split line by line; and determining the data corresponding to the split field based on the field separator corresponding to the split field.

[0054] In an optional embodiment, when using MapReduce to perform the splitting process, the data file to be split is read line by line, the field delimiters of the split fields are used, the line by line data is divided into several fields, and then data verification is performed on each field.

[0055] Optionally, in the above embodiment of the present invention, after splitting the file to be split based on the split field to obtain multiple sub-files, the method also includes: obtaining a first data volume of the multiple sub-files and a second data volume of the error file; obtaining a split ratio corresponding to the split field based on the first data volume and the second data volume; determining a target split field based on the split ratio, wherein the target split field is used for the next splitting of the file to be split.

[0056] In an optional embodiment, after the data file to be split is split using the automatically identified splitting field, a series of sub-files and an ERROR file can be generated. Further, MapReduce can be used to customize the output statistical information, automatically calculate the data volume of each sub-file (i.e., the first data volume mentioned above) and the data volume of the ERROR file (i.e., the second data volume mentioned above), and the split ratio of the data file using the currently automatically identified splitting field can be calculated accordingly, wherein the calculation formula of the split ratio S is as follows:

[0057]

[0058] Among them, M i Indicates the data size of the ith sub-file, M error Indicates the data size of the ERROR file.

[0059] Optionally, in the above embodiment of the present invention, when there are multiple split fields, the file to be split is split based on the multiple split fields, the split ratio corresponding to each split field is determined, and the split field corresponding to the highest split ratio is determined as the target split field.

[0060] In an optional embodiment, in the case where there are multiple splitting fields that can be used to split the file to be split, multiple splitting operations can be performed on the file to be split according to the multiple splitting fields, and a comparative test can be performed. Based on the splitting ratio obtained from each splitting operation, the field corresponding to the highest splitting ratio is determined as the target splitting field for subsequent use.

[0061] In another optional embodiment, the splitting ratio obtained after the above splitting is performed according to the automatically identified splitting field can be customized, and the user can customize multiple splitting fields for splitting, thereby solving the situation where there is still no data quality issue but the data cannot be split after the target splitting field is determined.

[0062] Combine the following Figure 2 and Figure 3 A preferred embodiment of the present invention is described in detail.

[0063] like Figure 2As shown, for MapReduce, the entire system architecture may include: ORALCE server, receiving server, splitting cluster, download server, and ETL (Extract-Transform-Load) job scheduling. Among them, the receiving server is used to receive the files to be split and the splitting rule files uploaded by the user; the ETL job scheduling is used to schedule the receiving server to store the received files in the splitting cluster, and schedule the splitting cluster to perform the file splitting operation in parallel; the ORALCE server is used as a configuration library, mainly for storing splitting rules, splitting method configuration, data verification rules, etc.; the splitting cluster is used to split the files to be split; the download server is used to receive the file download request sent by the user, and merge the sub-files corresponding to the request for compression transcoding (that is, encoding format conversion), and finally send the file to the client.

[0064] For split clusters, such as Figure 2 and Figure 3 As shown, the file splitting process is as follows: read the splitting configuration information from the database according to the splitting job (which can be the work processing unit completed by the computer required for the file splitting task), and obtain all the fields of the data file to be split in this splitting job; automatically identify the splitting field matching splitting rules according to the fields of the data file to be split; read the data file and perform data verification according to all the fields of the file to be split; after passing the data verification, execute the main splitting process according to the automatically identified splitting fields and the matching splitting rules; split the data file to be split into multiple sub-files, and write the data with data quality problems and the data that does not match the splitting rules in the data verification process to the ERROR file; count the data volume of all splitting result files and ERROR files respectively; calculate the splitting ratio according to the splitting result data volume and the actual data volume of the data file. Process the splitting rule data (Key / Value) according to certain specific rule files (containing the rule data that other file splitting depends on) and store them in HBase. Moreover, filter and archive the ERROR file to facilitate the subsequent problem troubleshooting to find out the reasons for the failure to split.

[0065] Through the above scheme, the splitting efficiency is improved and the splitting time is shortened by adopting a distributed computing method; the problem of manual input is avoided by automatic identification of split fields, which improves work efficiency and splitting accuracy; the data is processed line by line during the splitting process, and the data with data quality problems is filtered according to the configured data verification rules to improve data quality and availability; the personalized configuration of splitting rules is supported to achieve customizable splitting to meet different business needs; for data files with consistent splitting rules, only the relevant information of the splitting needs to be configured without modifying the source code, which improves the code reuse rate and reduces the development workload; the automatic statistics of the splitting result data volume and the splitting ratio are calculated, which greatly reduces the cycle of manual verification of the splitting results and adjustment of the splitting rules, and greatly improves work efficiency; the multi-field combination splitting makes the data file splitting more thorough, which reduces the situation where the splitting result file data is incomplete due to incomplete splitting, resulting in inconsistency with the front-end data when the business uses the data.

[0066] Example 2

[0067] According to an embodiment of the present invention, a file processing device is also provided, which can execute the file processing method in the above embodiment. The specific implementation scheme and preferred application scenario are the same as those in the above embodiment and will not be described in detail here.

[0068] Optionally, the method may be deployed in a processing node of a distributed processing cluster.

[0069] The above-mentioned distributed processing cluster can be a Hadoop cluster. MapReduce in the Hadoop cluster is a parallel computing model, framework and platform for large-scale data processing. The processing node can be a node in the Hadoop cluster. Each node can be deployed on a server, or multiple nodes can be deployed on the same server.

[0070] In an optional embodiment, a distributed splitting method can be used to achieve the purpose of splitting a large number of files at the same time, thereby greatly improving the splitting efficiency.

[0071] Figure 4 is a schematic diagram of a file processing device according to an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0072] An acquisition module 42 is used to acquire a file to be split and a splitting rule corresponding to the file to be split, wherein the file to be split includes multiple fields;

[0073] An identification module 44, used for identifying multiple fields in the file to be split;

[0074] A matching module 46, used for matching the multiple fields with the splitting rule to determine the splitting field among the multiple fields;

[0075] The splitting module 48 is used to split the file to be split based on the splitting field to obtain multiple sub-files.

[0076] Optionally, in the above embodiment of the present invention, the matching module 46 includes: a generation unit for generating a regular expression based on a splitting rule; a matching unit for matching multiple fields with the regular expression; and a first determination unit for determining that a field that successfully matches the regular expression is a splitting field.

[0077] Optionally, in the above embodiment of the present invention, the device also includes: an acquisition module 42 is also used to obtain data verification rules, and data corresponding to the split fields in the file to be split; a verification module is used to verify the data corresponding to the split fields based on the data verification rules; the splitting module 48 is also used to split the file to be split based on the split fields if the verification is successful; and a storage module is used to store the data corresponding to the split fields in an error file if the verification fails.

[0078] Optionally, in the above embodiment of the present invention, the acquisition module 42 includes: a reading unit, used to read the data in the file to be split line by line; and a second determination unit, used to determine the data corresponding to the split field based on the field separator corresponding to the split field.

[0079] Optionally, in the above embodiment of the present invention, the device also includes: an acquisition module 42 is also used to obtain the first data volume of multiple sub-files and the second data volume of the error file; a processing module is used to obtain the split ratio corresponding to the split field based on the first data volume and the second data volume; a determination module is used to determine the target split field based on the split ratio, wherein the target split field is used to split the file to be split next time.

[0080] Optionally, in the above embodiment of the present invention, when there are multiple split fields, the splitting module 48 is also used to split the file to be split based on multiple split fields; the acquisition module 42 is also used to determine the split ratio corresponding to each split field, and determine the split field corresponding to the highest split ratio as the target split field.

[0081] Example 3

[0082] According to an embodiment of the present invention, a file processing system is also provided. The device can execute the file processing method in the above embodiment. The specific implementation scheme and preferred application scenario are the same as the above embodiment and will not be described in detail here.

[0083] Figure 5 is a schematic diagram of a file processing system according to an embodiment of the present invention. Figure 5 As shown, the system includes:

[0084] The database 52 is used to store the splitting rules corresponding to the files to be split, wherein the files to be split contain multiple fields.

[0085] The above database can be a relational database or a distributed storage system, wherein the relational database can be Oracle, MySQL or DB2, etc., but not limited thereto; the distributed storage system can be HBase, which is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. The powerful reading capability provided by HBase can meet the business scenario of frequently reading rule data (Key / Value) for associated splitting during the splitting process.

[0086] The distributed processing cluster 54 is connected to the database and includes processing nodes. The distributed processing cluster is used to store the files to be split. The processing nodes are used to identify multiple fields in the files to be split, match the multiple fields with the splitting rules, determine the splitting fields in the multiple fields, and split the files to be split based on the splitting fields to obtain multiple sub-files.

[0087] The above-mentioned distributed processing cluster can be a Hadoop cluster. MapReduce in the Hadoop cluster is a parallel computing model, framework and platform for large-scale data processing. The processing node can be a node in the Hadoop cluster. Each node can be deployed on a server, or multiple nodes can be deployed on the same server.

[0088] Optionally, in the above embodiment of the present invention, the processing node is further used to generate a regular expression based on the splitting rule, match multiple fields with the regular expression, and determine the field that successfully matches the regular expression as the splitting field.

[0089] Optionally, in the above embodiment of the present invention, the processing node is also used to obtain data verification rules, and the data corresponding to the split fields in the file to be split, and verify the data corresponding to the split fields based on the data verification rules; wherein, if the verification is successful, the file to be split is split based on the split fields; if the verification fails, the data corresponding to the split fields are stored in an error file.

[0090] Optionally, in the above embodiment of the present invention, the processing node is further used to read the data in the file to be split line by line, and determine the data corresponding to the split field based on the field separator corresponding to the split field.

[0091] Optionally, in the above embodiment of the present invention, the processing node is also used to obtain the first data volume of multiple sub-files and the second data volume of the error file, obtain the split ratio corresponding to the split field based on the first data volume and the second data volume, and determine the target split field based on the split ratio, wherein the target split field is used to split the file to be split next time.

[0092] Optionally, the processing node is further used to split the file to be split based on multiple split fields when there are multiple split fields, determine the split ratio corresponding to each split field, and determine the split field corresponding to the highest split ratio as the target split field.

[0093] Example 4

[0094] According to an embodiment of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the file processing method in the above-mentioned embodiment 1.

[0095] Example 5

[0096] According to an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the file processing method in the above-mentioned embodiment 1 is executed when the program is run.

[0097] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0098] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A file processing method, characterized in that: The method is applied to a processing node of a distributed processing cluster, wherein the method comprises: Obtain a file to be split and a splitting rule corresponding to the file to be split, wherein the file to be split contains multiple fields; Identifying the multiple fields in the file to be split; Matching the multiple fields with the splitting rule to determine the splitting fields among the multiple fields; Splitting the to-be-split file based on the splitting field to obtain a plurality of sub-files; Before the file to be split is split based on the split field to obtain multiple sub-files, the method further includes: obtaining data verification rules and data corresponding to the split field in the file to be split; verifying the data corresponding to the split field based on the data verification rules; if the verification succeeds, the file to be split is split based on the split field; if the verification fails, the data corresponding to the split field is stored in an error file, wherein the error file represents a file for storing problematic data, wherein the data verification rules include one or more of data length, data type and numerical accuracy; Matching the multiple fields with the splitting rule to determine the splitting field among the multiple fields, including: generating a regular expression based on the splitting rule; matching the multiple fields with the regular expression; determining the field that successfully matches the regular expression as the splitting field; After the file to be split is split based on the split field to obtain multiple sub-files, the method further includes: obtaining a first data volume of the multiple sub-files and a second data volume of the error file; obtaining a split ratio corresponding to the split field based on the first data volume and the second data volume; and determining a target split field based on the split ratio, wherein the target split field is used to split the file to be split next time.

2. The method according to claim 1, characterized in that Acquiring data corresponding to the split fields in the to-be-split file includes: Read the data in the file to be split line by line; Based on the field separator corresponding to the split field, the data corresponding to the split field is determined.

3. The method according to claim 1, characterized in that: In the case where there are multiple split fields, the to-be-split file is split based on the multiple split fields, the split ratio corresponding to each split field is determined, and the split field corresponding to the highest split ratio is determined as the target split field.

4. A file processing device, characterized in that: The device is deployed in a processing node of a distributed processing cluster, wherein the device comprises: An acquisition module, used for acquiring a file to be split and a splitting rule corresponding to the file to be split, wherein the file to be split includes multiple fields; An identification module, used for identifying the multiple fields in the file to be split; A matching module, used for matching the multiple fields with the splitting rule to determine a splitting field among the multiple fields; A splitting module, used for splitting the to-be-split file based on the splitting field to obtain a plurality of sub-files; The splitting module is further used to obtain data verification rules and data corresponding to the split fields in the file to be split; based on the data verification rules, the data corresponding to the split fields are verified; if the verification succeeds, the file to be split is split based on the split fields; if the verification fails, the data corresponding to the split fields are stored in an error file, wherein the error file represents a file for storing problematic data, wherein the data verification rules include one or more of data length, data type and numerical accuracy; The matching module includes a generating unit, a matching unit and a first determining unit, wherein the generating unit is used to generate a regular expression based on the splitting rule; the matching unit is used to match the multiple fields with the regular expression; the first determining unit is used to determine that the field that successfully matches the regular expression is the splitting field; The device is also used to obtain a first data volume of the multiple sub-files and a second data volume of the error file after splitting the file to be split based on the split field to obtain multiple sub-files, obtain a split ratio corresponding to the split field based on the first data volume and the second data volume, and determine a target split field based on the split ratio, wherein the target split field is used for splitting the file to be split next time.

5. A file processing system, characterized in that: include: A database, used to store splitting rules corresponding to a file to be split, wherein the file to be split contains multiple fields; A distributed processing cluster connected to the database, comprising processing nodes, the distributed processing cluster is used to store the to-be-split file, the processing nodes are used to identify the multiple fields in the to-be-split file, match the multiple fields with the splitting rules, determine the splitting fields among the multiple fields, and split the to-be-split file based on the splitting fields to obtain multiple sub-files; Wherein, the processing node is also used to obtain data verification rules and the data corresponding to the split fields in the file to be split; based on the data verification rules, the data corresponding to the split fields are verified; if the verification is successful, the file to be split is split based on the split fields; if the verification fails, the data corresponding to the split fields are stored in an error file, wherein the error file represents a file used to store problem data.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the file processing method according to any one of claims 1 to 3.

7. A processor, characterized in that: The processor is used to run a program, wherein the program executes the file processing method according to any one of claims 1 to 3 when running.

Images