Big data file processing system

Abstract

The invention relates to a big data file processing system, which comprises a rule configuration module, a data acquisition module, a file segmentation module, a data analysis module, a file cleaning module and a file merging module, and is characterized in that an analysis rule, a cleaning rule and a merging rule are pre-configured on a configuration platform; acquiring source reconciliation data, uploading the source reconciliation data to an object storage platform, configuring a thread in a data group form to read the data, writing the data into the fragmented file, reading the data by taking the written fragmented file and a line number based on the fragmented file as a primary key, analyzing the line data based on the analysis rule, cleaning the line data based on a cleaning pair to obtain a cleaning target table, and storing the cleaned target table into a database; and cleaning the cleaning target table, generating fragmented files based on the cleaned cleaning target table, carrying out file merging, and outputting processed file data. By means of the data processing method and device, adaptive data processing can be achieved, and more efficient data cleaning and conversion are achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a large data file processing system. Background Art

[0002] When processing reconciliation files of multiple asset management institutions, especially large-scale data files, the reconciliation files may contain millions or even billions of records, and the data volume is measured in TB or even PB. Data processing of this scale places extremely high demands on the system's storage, computing and transmission capabilities. When there is insufficient memory, the system will crash or become very slow. The reading and writing speed of large files will be limited by IO performance, which will also cause the program to become very slow.

[0003] In addition, the reconciliation file formats, structures, and contents provided by different asset management institutions may differ significantly. This diversity requires the processing system to be highly flexible and scalable to support the parsing and conversion of multiple data formats. It is precisely because of the diversity of data formats that the processing logic cannot be shared, and the parsing and cleaning rules need to be redeveloped for new files each time.

[0004] If an exception occurs during file processing, it is impossible to identify which row or user's data is wrong, and all data can only be re-parsed and cleaned. Large file processing involves complex data segmentation, merging, conversion and other operations, which requires large amounts of calculation and low processing efficiency.

[0005] Currently, no effective solution has been proposed for the problems existing in the relevant technologies. Summary of the invention

[0006] The embodiment of the present application provides a big data file processing system for achieving efficient parsing and cleaning of very large files, reducing the memory usage and processing time of a single node, improving processing efficiency, and improving processing flexibility according to custom configuration.

[0007] The present application provides a large data file processing system, including:

[0008] A rule configuration module is used to pre-configure parsing rules, cleaning rules, and merging rules on the configuration platform, wherein the cleaning rules include data cleaning rules and file cleaning rules;

[0009] The data acquisition module is used to acquire source reconciliation data and upload it to the object storage platform. During the upload process, each time a file is uploaded, a file upload message is sent to the message queue. The file upload message includes: file name and file path;

[0010] A file segmentation module is used to consume the uploaded message to obtain the to-be-processed file, configure the to-be-processed file in the form of a data group to read the data through a thread, write the to-be-processed file into a shard file, and transfer the written shard file to the object storage platform. Each time a shard file is uploaded, a shard upload message is sent to the message queue, and the shard upload message includes: the file name and file path of the shard file;

[0011] A data parsing module is used to consume the shard upload message to obtain a shard file from the object storage platform, read data based on the row number of the shard file as the primary key and parse the row data based on the parsing rule, and re-transmit the parsed row data formed by the parsed row data and the primary key to the object storage platform, and send a parsing message to the message queue after each shard file parsing is completed, wherein the parsing message includes: the maximum value of the global primary key, the maximum value of the current shard primary key and its minimum value, and the parsing rule is configured to include a data delimiter and the number of row data fields;

[0012] A file cleaning module, used for consuming the parsed message to obtain a shard file from the object storage platform, cleaning the row data based on the data cleaning rule to obtain a cleaning target table, then cleaning the cleaning target table based on the file cleaning rule, generating a shard file based on the cleaned cleaning target table and uploading it to a to-be-merged file directory of the object storage platform;

[0013] The file merging module is used to merge the fragment files of the to-be-merged file directory based on the merging rule and output the processed file data.

[0014] In some embodiments, the file segmentation module further comprises:

[0015] Initialize the read thread and write thread group;

[0016] The reading thread reads the data of the to-be-processed file line by line, and puts n lines of data into an ordered queue as a data group, and the data groups are marked with group numbers;

[0017] Multiple writing threads of the writing thread group take out data groups from the ordered queue, and write every m data groups into a shard file. During the writing process, the line number of each data in the file to be processed is calculated according to the group number of the current data group and the maximum line number of each shard file is recorded. n and m are pre-configured natural numbers.

[0018] In some embodiments, the file cleaning module further includes:

[0019] A data cleaning module, used for consuming the parsed message to obtain a shard file from the object storage platform, reading the shard file according to the primary key and cleaning the row data based on the data cleaning rule, saving the cleaned data to the cleaning target table, and sending a data cleaning message to the message queue after each row of data is cleaned. The data cleaning rule is configured as a cleaning method for data fields in the row of data;

[0020] The target table cleaning module is used to consume the data cleaning message, query the cleaning target table based on the file cleaning rule, generate a fragment file, and upload it to the object storage platform, configure a target file cleaning counter, and each time a fragment file is generated, the target file cleaning counter counts 1. The file cleaning rule is configured as the data screening condition required to generate the fragment file.

[0021] In some of the embodiments, the file merging module adopts an automatic merging strategy and / or a fixed-point triggering strategy to merge files.

[0022] In some of the embodiments, when an automatic merge strategy is used to merge files,

[0023] Monitor the directory of files to be merged in real time. When a segmented file is passed into the directory of files to be merged, trigger the file download task and download the incoming segmented file to the local system.

[0024] Get the file name of the shard file, sort the shard files according to the merging rule, merge adjacent shard files and write them into a temporary file. When the number of records in the temporary file is equal to the maximum row number, the file merge is completed. The merging rule includes: the merging order and merging object of the shard files.

[0025] In some of the embodiments, when a fixed-point starting strategy is used to merge files, a fixed-point trigger scheduling service is configured, and the fixed-point trigger scheduling service re-merges the files based on a preset target file and a merge file rule.

[0026] In some embodiments, in the data parsing module, if an error occurs while parsing the current row of data, the current error data is recorded in an error table, and parsing of the next row of data continues, wherein the error data in the error table is reprocessed by configuring a scheduling service to query the error table, and the scheduling service at least includes querying the error table, processing error records, and updating status.

[0027] In some embodiments, in the data cleaning module, if an error occurs while cleaning the current row of data, the current error data is recorded in an error table, and cleaning of the next row of data continues, wherein the error data in the error table is reprocessed by configuring a scheduling service to query the error table, and the scheduling service at least includes querying the error table, processing error records, and updating status.

[0028] In some embodiments, the data parsing module is configured with a parsing counter.

[0029] If the current row of data in the shard file is parsed successfully, the parsing counter will be incremented by 1;

[0030] If the parsing of the current row of data is wrong and has been reprocessed by the scheduling service, the parsing counter is incremented by 1;

[0031] When the count value of the parsing counter is equal to the maximum line number, it is determined that the parsing of the current segment file is completed.

[0032] In some embodiments, the data cleaning module is configured with a cleaning counter.

[0033] If the current row of data in the shard file is successfully cleaned, the cleaning counter count increases by 1;

[0034] If there is an error in cleaning the current row of data and it has been reprocessed by the scheduling service, the cleaning counter count is increased by 1;

[0035] When the count value of the cleaning counter is equal to the maximum row number, it is determined that data cleaning of all row data of the current fragment file is completed.

[0036] Compared with related technologies, the big data file processing system provided in the embodiment of the present application can realize adaptive data processing and achieve more efficient data cleaning and conversion; it uses parallel computing technology to quickly parse and clean large files, and custom configurations can be added during the parsing and cleaning process, and the order of parsing and cleaning files can be manipulated at any time; it uses a distributed computing architecture to realize block parsing, distributed machine parsing, multi-machine parallel computing, and parallel merging of large files.

[0037] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a structural block diagram of a big data file processing system according to an embodiment of the present application;

[0040] Figure 2 is a substructure block diagram of a big data file processing system according to an embodiment of the present application;

[0041] Figure 3It is a schematic diagram of the principle of a big data file processing system according to a preferred embodiment of the present application.

[0042] In the figure:

[0043] 1. Rule configuration module; 2. Data acquisition module; 3. File segmentation module;

[0044] 4. Data analysis module; 5. File cleaning module; 6. File merging module;

[0045] 51. Data cleaning module; 52. Target table cleaning module. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0047] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0048] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a kind of", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices.

[0050] The present embodiment provides a large data file processing system. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated. Figure 1-Figure 2 This is a structural block diagram of a large data file processing system according to an embodiment of the present application. Figure 1-Figure 2 A detailed introduction to the various components of the big data file processing system:

[0051] Rule configuration module 1 is used to pre-configure parsing rules, cleaning rules, and merging rules on the configuration platform, where cleaning rules include data cleaning rules and file cleaning rules. If new file parsing needs to be added, parsing rules and cleaning rules can be automatically added without any code development. Through rule configuration module 1, users can easily add new parsing rules, cleaning rules, and merging rules according to their needs without any code development. This flexibility enables the system to quickly adapt to different data sources and file formats, supporting rapid expansion and changes in business.

[0052] Data acquisition module 2 is used to acquire source reconciliation data and upload it to the object storage platform. During the upload process, a file upload message is sent to the message queue for each file uploaded. The source object data is acquired from various forms of data sources, such as payment platforms, e-commerce platforms, POS systems, etc. The source reconciliation data is directly pulled from the data source and uploaded to the object storage platform. This enables the system to handle multiple data sources and file formats to meet the needs of different business scenarios. The file upload message includes: file name and file path;

[0053] The file segmentation module 3 is used to consume the uploaded message to obtain the to-be-processed file, configure the thread to read the data of the to-be-processed file in the form of a data group, write the sliced ​​file, and transfer the written sliced ​​file to the object storage platform. Each time a slice file is uploaded, a slice upload message is sent to the message queue, and the slice upload message includes: the file name and file path of the slice file;

[0054] Data parsing module 4, used for consuming the shard upload message to obtain the shard file from the object storage platform, reading data based on the row number of the shard file as the primary key and parsing the row data based on the parsing rule, re-transmitting the parsed row data and the primary key into the object storage platform, and sending a parsing message to the message queue after each shard file parsing is completed, wherein the parsing message includes: the maximum value of the global primary key, the maximum value of the current shard primary key and its minimum value, and the parsing rule is configured to include a data delimiter and the number of row data fields;

[0055] A file cleaning module 5 is used to consume the parsed message to obtain a shard file from the object storage platform, clean the row data based on the data cleaning rule to obtain a cleaning target table, then clean the cleaning target table based on the file cleaning rule, generate a shard file based on the cleaned cleaning target table, and upload it to the to-be-merged file directory of the object storage platform;

[0056] The file merging module 6 is used to merge the fragmented files of the to-be-merged file directory based on the merging rule and output the processed file data. Optionally, the file merging module 6 adopts an automatic merging strategy and / or a fixed-point trigger strategy to merge files.

[0057] It should be noted that the embodiment of the present application uses parallel computing to execute the above-mentioned file segmentation module 3, data analysis module 4 and file cleaning module 5, which greatly reduces the load pressure of a single node, improves the scalability and stability of the processing system, and also improves the processing efficiency, thereby greatly shortening the time required to process large files, improving processing efficiency, and saving costs and energy.

[0058] In some embodiments, the file segmentation module 3 further comprises:

[0059] Initialize the read thread and write thread group;

[0060] The reading thread reads the data of the to-be-processed file line by line, and puts n lines of data into an ordered queue as a data group, wherein the data groups are marked with group numbers, and the data group format may be a list;

[0061] Multiple writing threads of the writing thread group take out data groups from the ordered queue, and write every m data groups into a shard file. During the writing process, the line number of each data in the file to be processed is calculated according to the group number of the current data group and the maximum line number of each shard file is recorded. n and m are pre-configured natural numbers and can be flexibly configured. Their specific values ​​are not limited here.

[0062] Based on the above configuration, the file segmentation module 3 of this embodiment realizes efficient processing of large data files by reading and writing data through multi-threading. This can not only reduce the time required to process a single file, but also make full use of system resources and improve overall processing performance; at the same time, by segmenting a large file into multiple small fragment files, the system optimizes the storage and access performance of data, and the fragment files are more easily processed in parallel, and the storage and computing resources can be flexibly adjusted according to needs.

[0063] In some embodiments, the file cleaning module 5 further includes:

[0064] The data cleaning module 51 is used to consume the parsed message to obtain the shard file from the object storage platform, read the shard file according to the primary key and clean the row data based on the data cleaning rule, save the cleaned data to the cleaning target table, and send a data cleaning message to the message queue every time the cleaning of a row of data is completed. The data cleaning rule is configured as a cleaning method for the data field in the row data, such as performing data type conversion, addition, subtraction, multiplication and division mathematical operations on a certain source data field, or using it as an input parameter to call the cleaning interface to achieve complex cleaning;

[0065] The target table cleaning module 52 is used to consume the data cleaning message, query the cleaning target table based on the file cleaning rule, generate a shard file, and upload it to the object storage platform, configure a target file cleaning counter, and each time a shard file is generated, the target file cleaning counter counts 1, and the file cleaning rule is configured to generate the data screening conditions required for the shard file, such as selecting to generate the cleaning data of a certain IOU number segment according to the IOU number range, or selecting the range according to the primary key segment. Taking the selection of generating the cleaning data of a certain IOU number segment according to the IOU number range as an example, the file directory of the generated shard file is configured as: / source file name / shard file name; the format of the shard file name is configured as: source file name_[segment type]_[segment start]_[segment end]_[number of records], and the source file name examples are as follows: loan_loan_100_200_100, loan_loan_200_300_100, and the file content includes the head but not the tail.

[0066] The above embodiment cleans and verifies the original data through pre-configured cleaning rules, thereby ensuring the quality and accuracy of the data, which helps to improve the accuracy and reliability of subsequent analysis, decision-making and application, thereby improving the overall value of the data.

[0067] In the above embodiment, when the system adopts the automatic merging strategy to merge files,

[0068] Monitor the directory of files to be merged in real time. When a segmented file is passed into the directory of files to be merged, trigger the file download task and download the incoming segmented file to the local system.

[0069] Get the file name of the shard file, sort the shard files according to the merging rule, merge the adjacent shard files and write them into a temporary file. When the number of records in the temporary file is equal to the maximum line number, the file merge is completed. The merging rule includes: the merging order of the shard files and the merging objects; the merging objects can be of the same type or different types, such as merging the shard files of a certain loan number segment and the shard files of a certain contract number segment; during the merging process, the merging track should be recorded, such as the file names and number of records before and after the merging, and a merged file and a to-be-merged file can also be maintained. The list or index of merged files distinguishes merged files and files with merge, ensuring that merged files will not be merged repeatedly. The temporary file after the merger is named: source file name_[segment type]_[segment start]_[segment end]_[number of records]. For example, after loan_loan_100_200_100 and loan_loan_200_300_100 are merged, a new file loan_loan_100_300_200 is generated. During the merging process, each time a record is successfully written, the corresponding merged file cleaning counter is increased by 1.

[0070] In some of the embodiments, when a fixed-point departure strategy is used to merge files, a fixed-point trigger scheduling service is configured on a scheduling platform, such as Apache Airflow, Luigi, Azure Data Factory, AWS Step Functions, etc. The fixed-point trigger scheduling service re-merges files based on preset target files and merge file rules.

[0071] Based on this fixed-point dispatching service, when the files merged by the automatic merge strategy do not meet the requirements or the files are deleted by mistake, repairs are performed to ensure that the merge logic is flexible enough, which helps to discover and solve problems in a timely manner.

[0072] In some embodiments, in the data parsing module 4, if an error occurs in parsing the current row of data, the current error data is recorded in an error table, and parsing of the next row of data continues, wherein the error data in the error table is reprocessed by configuring a scheduling service to query the error table, and the scheduling service at least includes querying the error table, processing error records, and updating status.

[0073] In some embodiments, in the data cleaning module 51, if an error occurs in cleaning the current row of data, the current error data is recorded in an error table, and cleaning of the next row of data continues, wherein the error data in the error table is reprocessed by configuring a scheduling service to query the error table, and the scheduling service at least includes querying the error table, processing error records, and updating status.

[0074] In some embodiments, the data parsing module 4 is configured with a parsing counter.

[0075] If the current row of data in the shard file is parsed successfully, the parsing counter will be incremented by 1;

[0076] If the parsing of the current row of data is wrong and has been reprocessed by the scheduling service, the parsing counter is incremented by 1;

[0077] When the count value of the parsing counter is equal to the maximum line number, it is determined that the parsing of the current segment file is completed.

[0078] In some embodiments, the data cleaning module 51 is configured with a cleaning counter.

[0079] If the current row of data in the shard file is successfully cleaned, the cleaning counter count increases by 1;

[0080] If there is an error in cleaning the current row of data and it has been reprocessed by the scheduling service, the cleaning counter count is increased by 1;

[0081] When the count value of the cleaning counter is equal to the maximum row number, it is determined that data cleaning of all row data of the current fragment file is completed.

[0082] In the above embodiment, the count values ​​of the parsing counter, the cleaning counter, and the target file cleaning counter can be displayed through an interactive interface to obtain the file processing progress in real time, analyze the processing situation, and take early warning or adjustment measures.

[0083] Based on the above embodiment, the system transmits the processing status and results between each module through the message queue, so that the entire processing flow can be traced. At the same time, each module retains detailed log information, which is convenient for subsequent problem troubleshooting and auditing. With modular design, each module is relatively independent and easy to maintain and upgrade. When it is necessary to add new functions or fix bugs, operations can be performed only on related modules without affecting the stability and performance of the entire system.

[0084] The entire processing flow (from data acquisition to file merging) is highly automated, reducing the need for manual intervention, reducing operational complexity and error rates. At the same time, through pre-configured rules, the system can intelligently parse, clean and merge data, improving processing efficiency and accuracy.

[0085] Data in multiple modules is stored on the object storage platform, ensuring data security and reliability. Even if a processing module fails, data will not be lost, and the entire process can be ensured to proceed smoothly by reprocessing or restoring data.

[0086] The embodiments of the present application are described and illustrated below through preferred embodiments. Figure 3 is a schematic diagram of the principle of a large data file processing system according to a preferred embodiment of the present application, with reference to Figure 3 As shown, the system includes the following processing steps:

[0087] 1. System configuration: Configure the parsing rules, cleaning rules, and merging rules for each file on the configuration platform. Figure 3 In the process of configuring the platform configuration service, you can pull the configuration in the subsequent process of parsing services, cleaning services, generating target shard files, and merging shard files.

[0088] 2. Get source reconciliation to the object storage platform: You can use the pull service to obtain reconciliation files from various sources, directly pull and upload them to the object storage platform, and send an MQ1 message for each file uploaded. The message content includes the file name and path.

[0089] 3. Pull the file, split it and fill in the line number: Start the preprocessing service, consume the MQ1 message to pull the corresponding file, recorded as F, start the reading thread and the writing thread group, the reading thread reads line by line, and each n data is a group, such as 100 or 1000, etc. For each group of data list read, record the group number, and put it into the ordered queue according to the group number. Multiple threads of the writing thread group obtain the data group from the ordered queue and write a new shard file to be cleaned. m data groups can be merged into one file, such as 5 or 10. When writing the shard file, according to the current data group number, calculate the real line number relative to the source file F, and save the maximum line number into the database. The written file is recorded as Fn, and Fn is uploaded to the object storage platform. Send an MQ2 message, and the message content includes the file name and path.

[0090] 4. Pull the shard file and parse it into the database: consume MQ2 messages through the parsing service, obtain the shard file from the object storage platform, set the row number as the primary key, read the data in sequence, and pull the parsing rules of the configuration platform. The parsing rules include data delimiters, the number of row data fields, etc. The parsed row data is stored as a new row of records with the primary key. After each shard file is processed, an MQ3 message is sent out. The message content includes the maximum and minimum values ​​of the current shard primary key, as well as the maximum value of the global primary key. If an error occurs in parsing a row, the current record is recorded in the error table and other data is processed. During the parsing process, the parsing counter is increased by 1 for each successful record. For the error data, the scheduling service queries the error table for reprocessing, and the parsing counter is increased by 1 for each successful record. When the parsing counter is equal to the maximum row number in the third step. The parsing stage is completed.

[0091] 5. Pull the shard files and clean them into the warehouse: Consume MQ3 messages through the cleaning service, query a certain section of data according to the primary key, query the cleaning data rules of the corresponding pull configuration platform, and clean each row of data according to the cleaning rules. The specific fields to be cleaned can be configured on the configuration platform. For example, if there are 10 fields in a row, you can configure the cleaning of the 5th field and the cleaning rules. The cleaning rules can be simple data type conversion of the source field, addition, subtraction, multiplication and division, or as input parameters, calling other interfaces to achieve more complex cleaning. The new records after cleaning are saved to the cleaning target table. If an error occurs when cleaning a row, the current record is recorded in the error table and other data continues to be processed. During the cleaning process, the cleaning counter is increased by 1 for each successful record. For the error data, the scheduling service queries the error table for reprocessing, and the cleaning counter is increased by 1 for each successful record. When the count value of the cleaning counter is equal to the maximum row number in step 3, the cleaning stage is completed and the MQ4 message is sent out.

[0092] 6. Generate target shard files: Consume MQ4 messages, query the cleaning file rules of the configuration platform. The rules can be configured to filter data according to a certain range of a field to generate a range of data files. For example, you can select to generate cleaning data for a certain IOU number range according to the IOU number range, or you can filter the range by segmentation according to the primary key of the parsed table. Query the cleaning target table according to the rules, generate the target shard files, and upload them to the object storage platform. During the generation process, each successful record is written, and the corresponding target file cleaning counter +1.

[0093] 7. Merge the target shard files: Two strategies are adopted, one is automatic merging and the other is fixed-point triggering.

[0094] Automatic merging: monitor the file directory specific to the file storage platform. As long as there is a file in it, download the file locally and query the merging rules of the configuration platform. The rules can configure the merging order and merging objects of the shard files. The same type can be merged or different types can be merged. For example, the shard files of a certain loan number segment can be merged with the shard files of a certain contract number segment, and they can be combined as needed. Get the file name and sort it according to the configuration rules to determine whether the shards are adjacent. Both adjacent shards can be merged, and the merge track is recorded. The merged files will not be merged repeatedly. The merged temporary file is named source file name_[segment type]_[segment start]_[segment end]_[number of records]. For example, after loan_loan_100_200_100 and loan_loan_200_300_100 are merged, a new file loan_loan_100_300_200 is generated. And so on, until the number of records in the last generated temporary file is equal to the number of records in the source file, the merge is completed, and the target accounting file is obtained. During the merging process, each time a record is successfully written, the corresponding merged file cleaning counter is increased by 1.

[0095] Fixed-point trigger: If you are not satisfied with the files generated by automatic merging or the files generated by automatic merging are deleted by mistake, you can configure a fixed-point trigger on the scheduling platform to merge a certain file again.

[0096] During the whole process, the values ​​of multiple counters can be monitored and displayed in real time.

[0097] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0098] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A big data file processing system, characterized in that: include: A rule configuration module is used to pre-configure parsing rules, cleaning rules, and merging rules on the configuration platform, wherein the cleaning rules include data cleaning rules and file cleaning rules; The data acquisition module is used to acquire source reconciliation data and upload it to the object storage platform. During the upload process, each time a file is uploaded, a file upload message is sent to the message queue. A file segmentation module is used to consume the uploaded message to obtain the file to be processed, configure the thread to read the data of the file to be processed in the form of a data group, write the slice file, and transfer the written slice file to the object storage platform, and send a slice upload message to the message queue every time a slice file is uploaded; A data parsing module is used to consume the shard upload message to obtain the shard file from the object storage platform, read the data based on the row number of the shard file as the primary key and parse the row data based on the parsing rule, and re-transmit the parsed row data formed by the parsed row data and the primary key to the object storage platform, and send a parsing message to the message queue after each shard file parsing is completed; A file cleaning module, used for consuming the parsed message to obtain a shard file from the object storage platform, cleaning the row data based on the data cleaning rule to obtain a cleaning target table, then cleaning the cleaning target table based on the file cleaning rule, generating a shard file based on the cleaned cleaning target table and uploading it to a to-be-merged file directory of the object storage platform; The file merging module is used to merge the fragment files of the to-be-merged file directory based on the merging rule and output the processed file data.

2. The big data file processing system according to claim 1, characterized in that: The file segmentation module further comprises: Initialize the reading thread and writing thread group; The reading thread reads the data of the to-be-processed file line by line, and puts n lines of data into an ordered queue as a data group, and the data groups are marked with group numbers; Multiple writing threads of the writing thread group take out data groups from the ordered queue, and write every m data groups into a shard file. During the writing process, the line number of each data in the file to be processed is calculated according to the group number of the current data group and the maximum line number of each shard file is recorded. n and m are pre-configured natural numbers.

3. The big data file processing system according to claim 2, characterized in that: The file cleaning module further includes: A data cleaning module is used to consume the parsed message to obtain a shard file from the object storage platform, read the shard file according to the primary key and clean the row data based on the data cleaning rule, save the cleaned data to the cleaning target table, and send a data cleaning message to the message queue after each row of data is cleaned; The target table cleaning module is used to consume the data cleaning message, query the cleaning target table based on the file cleaning rule, generate a shard file, and upload it to the object storage platform.

4. The big data file processing system according to claim 1, characterized in that: The file merging module adopts an automatic merging strategy and / or a fixed-point triggering strategy to merge files.

5. The big data file processing system according to claim 4, characterized in that: When using the automatic merge strategy to merge files, Monitor the directory of files to be merged in real time. When a segmented file is passed into the directory of files to be merged, trigger the file download task and download the incoming segmented file to the local system. Get the file name of the shard file, sort the shard files according to the merging rule, merge adjacent shard files and write them into a temporary file. When the number of records in the temporary file is equal to the maximum row number, the file merge is completed. The merging rule includes: the merging order and merging object of the shard files.

6. The big data file processing system according to claim 4, characterized in that: When using a fixed-point starting strategy to merge files, configure a fixed-point trigger scheduling service, which re-merges files based on the preset target files and merge file rules.

7. The big data file processing system according to claim 2, characterized in that: In the data parsing module, if an error occurs when parsing the current row of data, the current error data is recorded in an error table, and the next row of data is parsed, wherein the error data in the error table is reprocessed by configuring a scheduling service to query the error table.

8. The big data file processing system according to claim 3, characterized in that: In the data cleaning module, if an error occurs when cleaning the current row of data, the current error data is recorded in an error table, and the next row of data is cleaned, wherein the error data in the error table is reprocessed by configuring a scheduling service to query the error table.

9. The big data file processing system according to claim 7, characterized in that: In the data analysis module, a analysis counter is configured. If the current row of data in the shard file is parsed successfully, the parsing counter will be incremented by 1; If the parsing of the current row of data is wrong and has been reprocessed by the scheduling service, the parsing counter is incremented by 1; When the count value of the parsing counter is equal to the maximum line number, it is determined that the parsing of the current segment file is completed.

10. The big data file processing system according to claim 8, characterized in that: In the data cleaning module, a cleaning counter is configured. If the current row of data in the shard file is successfully cleaned, the cleaning counter count increases by 1; If there is an error in cleaning the current row of data and it has been reprocessed by the scheduling service, the cleaning counter count is increased by 1; When the count value of the cleaning counter is equal to the maximum row number, it is determined that data cleaning of all row data of the current fragment file is completed.

Images