A data file archiving method and system

Abstract

The present invention relates to the field of data processing technology, and discloses a data file archiving method and system. The method comprises the following steps: establishing a streaming data processing platform and configuring archiving information; scanning the data files to be archived on the streaming data processing platform and mapping them into corresponding data files, check files and ddl files, and adding the data file records to be archived in the database table; reading the database table in real time, and when a new data file record to be archived appears, reading the new data file to be archived and recording the corresponding archiving information; comparing the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived, if they are the same, archiving the data file corresponding to the new data file record to be archived and storing it in a storage medium, otherwise reporting an error. The present invention can realize the automation of the entire process of archiving historical data, avoiding the problems of long archiving time and data delay caused by human operation interference and large amount of data accumulation.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data file archiving method and system. Background Art

[0002] As banking business continues to develop and database systems continue to operate, a large amount of historical data (including files, database records, logs, etc.) will continue to accumulate. Among them, some data does not need to be accessed frequently, but still needs to be saved for backup. However, storing historical data and new data in the same database system will lead to problems such as insufficient disk space, performance degradation, and difficulty in data backup. Therefore, historical data archiving technology came into being. The main goal of historical data archiving technology is to move or copy data that does not need to be accessed frequently from the primary storage environment to the secondary storage environment (such as tapes, cloud storage, or lower-speed disks), thereby freeing up primary storage resources and improving the performance and scalability of the database system. In order to ensure data security and maximize data efficiency, banks store system data and external data in a unified manner in the data warehouse. When using data, the data to be used needs to be archived to the archiving platform for storage for processing and use.

[0003] Traditional historical data archiving technology uses batch archiving, which is done periodically or when the amount of historical data reaches a certain scale, and has low requirements for timeliness. This periodic task can only execute the archiving program at the scheduled time, and the data generated during the archiving time interval cannot be archived, resulting in the data in this time period being unusable; in addition, a large amount of historical data will accumulate during the archiving time interval, which takes a lot of time and occupies a lot of system resources (such as I / O resources) when archiving, affecting system performance. In addition, even if the traditional historical data archiving adopts the T+1 historical data archiving method, that is, archiving the data generated during the day at night, this data archiving method not only cannot meet the requirements of business scenarios for data timeliness, but is also prone to human operation errors. Summary of the invention

[0004] The purpose of the present invention is to provide an automated real-time historical data archiving method and system to solve the problems of traditional historical data archiving technology, such as delayed data utilization, long archiving time, large I / O resource occupation and high human operation error rate.

[0005] The present invention is achieved through the following technical solutions:

[0006] On the one hand, a data file archiving method is provided, comprising the following steps: establishing a streaming data processing platform on a data center or a cloud platform; configuring archiving information for each data file to be archived in a specified directory; the archiving information includes: archiving target type, archiving life cycle, archiving condition, file name and storage medium; performing the following steps on the streaming data processing platform: scanning all data files to be archived in the specified directory; mapping each scanned data file to be archived into a corresponding data file, check file and DDL file, and adding a corresponding data file record to be archived in a database table; the data file record to be archived includes dat a file, the path information of the check file and the path information of the ddl file; read the database table in real time, and when a new data file record to be archived appears in the database table, execute steps S1 and S2; wherein, step S1: read the archiving information corresponding to the new data file record to be archived; step S2: compare the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived according to the archiving condition in the archiving information, if they are the same, archive the data file corresponding to the new data file record to be archived and store it in the storage medium; if they are different, report an error and record the error in the log.

[0007] Furthermore, before reading the database table in real time, the method further includes the following steps: collecting data files to be archived from a data source; and caching the collected data files to be archived into a message queue.

[0008] Furthermore, after step S1, the following steps are also included: verifying whether the check file is consistent before and after archiving, verifying whether the data file is consistent before and after archiving, if the check file or the data file is inconsistent before and after archiving, interrupting the archiving process, if the check file and the data file are consistent before and after archiving, verifying whether the ddl file is consistent before and after archiving, if the newly added field is at the end, continuing the archiving process, if the newly added field is not at the end, interrupting the archiving process; creating a temporary table, and loading the data in the data file into the temporary table, comparing the number of data items in the temporary table after the data loading is completed with the number of data items in the check, and recording the data in the temporary table in the database table; writing the execution record into the log operation table of the database of the archiving system to generate an operation log.

[0009] Furthermore, the data file archiving process also includes the following steps: monitoring the archiving progress of data files; the archiving progress includes: archiving speed, archiving quantity, lost data files and duplicated data files; monitoring the storage medium status; the storage medium status includes: storage capacity, usage rate, hardware failure and network failure; if data files are lost during the data file archiving process, re-archiving or supplementing the lost data files; generating an archiving report; the archiving report includes archiving progress monitoring results, storage medium monitoring results and exception handling results; the exception handling results include: data file loss handling results, hardware failure handling results and network failure handling results.

[0010] Furthermore, after the data files are archived, the following steps are also included: backing up the archived data; the archived data backup includes: backup method, backup cycle, backup storage medium, backup data verification and backup recovery; the backup method includes: full backup, incremental backup or selected field backup; the backup storage medium includes: solid state drive; the backup data verification includes: readability verification and consistency verification; the backup recovery includes: providing data recovery when the stored data is damaged or the data file is unavailable.

[0011] On the other hand, a data file archiving system is provided, comprising:

[0012] A platform building module is used to build a streaming data processing platform on a data center or cloud platform;

[0013] An information configuration module is used to configure archiving information for each data file to be archived in a specified directory; the archiving information includes: archiving target type, archiving life cycle, archiving conditions, file name and storage medium;

[0014] A file scanning module, used to scan all the data files to be archived in the specified directory;

[0015] A file mapping module is used to map each scanned data file to be archived into a corresponding data file, a check file and a ddl file, and add a corresponding data file record to be archived in a database table; the data file record to be archived includes the path information of the data file, the path information of the check file and the path information of the ddl file;

[0016] A database table reading module is used to read the database table in real time, and when a new data file record to be archived appears in the database table, the file archiving module is controlled to work;

[0017] An archiving information reading module is used to read the archiving information corresponding to the new data file record to be archived;

[0018] A data analysis module is used to compare the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived according to the archiving condition in the archiving information, and if they are the same, archive the data file corresponding to the new data file record to be archived and store it in the storage medium; if they are different, report an error and record the error in a log;

[0019] A data collection module is used to collect data files to be archived from data sources;

[0020] The data cache module is used to cache the collected data files to be archived into the message queue.

[0021] The first verification module is used to verify whether the check file is consistent before and after archiving, and verify whether the data file is consistent before and after archiving. If the check file or the data file is inconsistent before and after archiving, the archiving process is interrupted. If the check file and the data file are consistent before and after archiving, the ddl file is verified before and after archiving. If the newly added field is at the end, the archiving process continues. If the newly added field is not at the end, the archiving process is interrupted.

[0022] The second check module is used to create a temporary table, load the data in the data file into the temporary table, compare the number of data entries in the temporary table after the data is loaded with the number of data entries in the check, and record the data in the temporary table into the database table;

[0023] The log generation module is used to write the execution record into the log operation table of the database of the archiving system to generate an operation log.

[0024] A first monitoring module is used to monitor the archiving progress of data files; the archiving progress includes: archiving speed, archiving quantity, lost data files and duplicated archived data files;

[0025] A second monitoring module is used to monitor the storage medium status; the storage medium status includes: storage capacity, usage rate, hardware failure and network failure;

[0026] A fault handling module is used to re-archive or supplement the lost data files when data files are lost during the data file archiving process;

[0027] The report generation module is used to generate an archiving report; the archiving report includes archiving progress monitoring results, storage medium monitoring results and exception handling results; the exception handling results include: data file loss handling results, hardware failure handling results and network failure handling results.

[0028] A data backup module is used to perform archived data backup; the archived data backup includes: backup method, backup cycle, backup storage medium, backup data verification and backup recovery; the backup method includes: full backup, incremental backup or selected field backup; the backup storage medium includes: solid state drive; the backup data verification includes: readability verification and consistency verification; the backup recovery includes: providing data recovery when the stored data is damaged or the data file is unavailable.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] Establish a streaming data processing platform in the data center or cloud platform of the enterprise or organization, and use streaming data processing technology to realize real-time archiving of historical data as soon as it enters the data warehouse. By configuring the streaming data processing platform and setting up an automated archiving process, production and business data can be archived in real time to the designated archiving storage medium according to the automated archiving process. This can realize the automation of the entire process of historical data archiving, avoid interference from human operations, improve the accuracy and reliability of data archiving, and avoid the long archiving time and data delay caused by the accumulation of a large amount of data. It breaks through the delay of traditional batch data processing, can make full use of the timeliness of data, and deeply mine and exert the business value of data. In addition, the use of real-time archiving avoids the system from performing a large number of I / O operations in a short period of time, and the overall system maintains high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of a data file archiving method flow chart provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0034] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is apparent to one of ordinary skill in the art that these specific details are not necessarily employed to practice the present invention. In other embodiments, in order to avoid obscuring the present invention, well-known structures, circuits, materials, or methods are not specifically described.

[0035] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "one embodiment," "an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. In addition, it will be appreciated by those of ordinary skill in the art that the figures provided herein are for illustrative purposes and that the figures are not necessarily drawn to scale. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0036] In the description of the present invention, the directions or positional relationships indicated by terms such as “front”, “rear”, “left”, “right”, “up”, “down”, “vertical”, “horizontal”, “high”, “low”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they should not be understood as limiting the scope of protection of the present invention.

[0037] Embodiment: Traditional historical data archiving generally adopts batch archiving method, that is, historical data within a certain time period is archived at a certain moment. This archiving method has a certain lag in data utilization; and, in order to save manpower and material resources, historical data archiving will centrally archive historical data within a certain time period. The amount of historical data archived centrally is large, and the archiving process takes a long time; in addition, in order not to affect the business, traditional historical data archiving is usually carried out at night, which requires a lot of time and energy, and is prone to human operational errors.

[0038] In view of the above-mentioned shortcomings of traditional historical data archiving, the first aspect of this embodiment provides a data file archiving method, which establishes a streaming data processing platform for receiving and storing production and business data on a data center or cloud platform of an enterprise or organization; then configures the streaming data processing platform, sets an automated archiving process, and archives the production and business data in real time to a designated archiving storage medium according to the automated archiving process. In addition, the method can also monitor abnormal conditions during the archiving process, complete automated processing, and establish data backup in the archiving storage medium.

[0039] The specific real-time process of this method is as follows Figure 1 As shown, the following steps are included:

[0040] Step 1: Establish a streaming data processing platform in a data center or cloud platform.

[0041] Among them, data centers are the core facilities of financial institutions, which are mainly responsible for storing, processing and managing the massive data of financial institutions, including customer information, transaction records, market analysis, etc. These data are crucial for the daily operations, risk management and decision support of financial institutions.

[0042] Cloud platforms are services based on hardware and software resources, which usually provide users with flexible and scalable IT resources in a virtualized manner. Cloud platforms provide a series of services such as computing, storage, networking, databases, and software development platforms through the Internet, allowing users to obtain and use these resources on demand without investing a lot of money to build and manage IT infrastructure. Cloud platforms have the advantages of elastic scaling, high availability, security, and metering and billing, providing enterprises with more flexible, efficient, and secure IT resource services. At present, most financial structures have established their own cloud platforms.

[0043] The streaming data processing platform is a system for real-time processing and analysis of streaming data. It relies on a data center or cloud platform and can process and analyze data collected in real time, but is not limited to it. The biggest advantage of the streaming data processing platform is that it breaks through the latency of traditional batch data processing, can make full use of the timeliness of data, and deeply mine and exert the business value of data. Data collection tools such as OGG and OMS are used to realize real-time data collection, and data is stored in real time through the data bus DataHub or data is cached using message queues. The streaming data computing engine Flink can analyze and process data in real time, and Hologres and MaxCompute big data warehouses are used to store data for analysis and decision-making. In addition, the streaming data processing platform can receive data in various forms, such as sensor data, log data, transaction data, etc.; and can clean, convert, filter and aggregate data to meet different business needs. The processed data needs to be stored in suitable storage such as tapes, disks (including hard disks and solid-state drives) and optical disks for subsequent analysis and query.

[0044] To establish a streaming data processing platform on a data center or cloud platform, please refer to the method for establishing the Alibaba Cloud streaming data processing platform. This embodiment establishes a streaming data processing platform and uses streaming data processing technology to archive historical data in real time as soon as it enters the data warehouse, thereby avoiding the long archiving time and data delay caused by a large amount of data accumulation; based on the streaming data processing platform, the entire process of historical data archiving is automated, thereby avoiding human operation interference and improving the accuracy and reliability of data archiving.

[0045] Step 2: Configure archiving information for each data file to be archived in the specified directory, and configure the port number of the source data end.

[0046] Specifically, the archiving information includes: archiving target type, archiving life cycle, archiving conditions, file name and storage medium. Among them, the archiving target type refers to the data classification type of the archiving target, which is divided into main file object, zipper object and detail file object. The archiving life cycle refers to the life cycle of the archived data, that is, the preservation duration, which is divided into daily preservation, weekly preservation, monthly preservation, annual preservation and permanent preservation. Archiving conditions refer to the conditions for executing archiving, which are divided into daily execution, end of month execution, end of ten days execution, end of quarter execution and end of year execution. The file name means that the name of the archived object must comply with the naming specifications and be executed in accordance with the bank's data governance specifications, generally system name_table name.

[0047] The purpose of configuring the archiving information of data files is to automate the entire process of historical data archiving, avoid human interference, and improve the accuracy and reliability of data archiving.

[0048] Step 3: Scan all the data files to be archived in the specified directory.

[0049] The streaming data processing platform relies on a data center or cloud platform and flexibly has multiple servers to support the running of its related programs. The related programs of the streaming data processing platform refer to all programs running on the streaming data processing platform, such as WeChat, QQ, DingTalk and other browsers. This step uses the scanning program of the streaming data processing platform to scan all files in the specified directory.

[0050] Step 4: Map each scanned data file to be archived into a corresponding data file, check file and ddl file, and add the corresponding data file record to be archived in the database table.

[0051] Specifically, the mapping program of the streaming data processing platform is used to map each data file to be archived into four corresponding files. The suffixes of these four files are .data, .check, .ddl and .ok, and the names of the three files are all table names, which refer to the data table names of the source. Among them, data represents data information, check represents verification information, ddl is a table creation statement, and ok indicates that the mapping is successful. After the mapping is successful, the streaming data processing platform records the path information of the data file, the path information of the check file, and the path information of the ddl file into the database table, forming a record corresponding to the data file to be archived.

[0052] Step 5: Read the database table in real time. When a new data file record to be archived appears in the database table, execute steps 6 and 7.

[0053] The database table in step 4 is read in real time using the reading program of the streaming data processing platform. When a new data file record to be archived appears in the database table, it means that a new data file needs to be archived.

[0054] Step 6: Read the new data file to be archived and record the corresponding archiving information.

[0055] Step 7: Compare the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived according to the archiving conditions in the archiving information. If they are the same, archive the data file corresponding to the new data file record to be archived and store it in the storage medium; if they are different, report an error and record the error in the log.

[0056] It should be noted that: in order to perform subsequent data file transfer operations, before reading the database table in real time, it is necessary to use the OGG data collection tool or MOS data collection tool of the streaming data processing platform to collect the data files to be archived from the data source (such as business system, database, file system, etc.); and cache the collected data files to be archived in the message queue of the streaming data processing platform.

[0057] This method uses streaming data processing technology to achieve real-time archiving of historical data as soon as it enters the data warehouse. Through the above steps 1 to 6, it can avoid the long archiving time and data delay caused by a large amount of data accumulation, realize the automation of the entire process of historical data archiving, and avoid human operation interference, thereby improving the accuracy and reliability of data archiving. In addition, the use of real-time archiving avoids the system from performing a large number of I / O operations in a short period of time, and the overall system maintains high efficiency.

[0058] In addition, in order to further improve the reliability of the entire data file archiving process, this method also performs file verification and archived data backup after completing the archiving of the current data file. The specific content of the file verification is as follows: after the current data file is archived, the data file and the check file are verified to be consistent before and after archiving. If the check file or the data file is inconsistent before and after archiving, the subsequent archiving process is interrupted. If the check file and the data file are consistent before and after archiving, the ddl file is verified to be consistent before and after archiving. If the newly added field is at the end, the archiving process continues. If the newly added field is not at the end, the subsequent archiving process is interrupted. It should be noted that: (1) The size, number of records and file name of the data file are recorded in the check file. If the size, number of records and file name of the data file recorded in the check file are inconsistent with the actual size, number of records and file name of the data file, the verification fails. (2) The ddl file records the column name and number of columns of the current archived file. The records of the ddl file are compared with the existing table structure in the archiving system. If the fields are consistent or there is a newly added field at the end, the archiving process continues. In other cases, the archiving process is interrupted. (3) Newly added fields refer to the data columns that are added in this archive compared to the previous archive; data columns refer to the multiple columns in each row of data in the data file separated by column delimiters, which is similar to the columns in an EXCEL table. (4) End refers to the newly added data columns at the end of the table; for example, in EXCEL, a new column is added at the far right, rather than in the middle.

[0059] In addition, a temporary table is created, and the data in the data file is loaded into the temporary table; after the data is loaded, the number of data in the temporary table is recorded, and the number of data in the temporary table after the data is loaded is compared with the number of data in the check. If the two numbers of data are the same, the data is loaded correctly, and the data in the temporary table is recorded in the database table, and the execution record (including: the name of the data file currently recorded in the database table, the execution time of recording the data in the temporary table in the database table, and the number of data in the database table after loading and other operation log data) is written into the log operation table of the database to generate an operation log. Archive data backup includes: backup method, backup cycle, backup storage medium, backup data verification and backup recovery; the backup method includes: full backup, incremental backup or selected field backup; the backup storage medium includes: solid state hard disk; the backup data verification includes: readability verification and consistency verification; the backup recovery includes: providing data recovery when the stored data is damaged or the data file is unavailable.

[0060] Furthermore, the method also automatically monitors the archiving progress and storage medium status during the archiving process, and automatically backs up the archived data. Specifically, the archiving progress monitoring includes monitoring indicators such as the archiving speed and the number of archives, as well as abnormal situations such as data file loss and repeated archiving of data files during the archiving process; the storage medium status monitoring includes monitoring indicators such as storage capacity, utilization rate, reliability, and fault conditions such as hardware failure and network failure. In the event that data files are lost during the data file archiving process, the lost data files are re-archived or supplemented through the streaming data processing platform. In order to facilitate the evaluation and optimization of the data file archiving process, the method generates an archiving report through the streaming data processing platform, and the archiving report includes the archiving progress monitoring results, the storage medium monitoring results, and the exception handling results; the exception handling results include: data file loss handling results, hardware failure handling results, and network failure handling results.

[0061] Corresponding to the data file archiving method provided in the first aspect above, the second aspect of this embodiment provides a data file archiving system, including:

[0062] A platform building module is used to build a streaming data processing platform on a data center or cloud platform;

[0063] An information configuration module is used to configure archiving information for each data file to be archived in a specified directory; the archiving information includes: archiving target type, archiving life cycle, archiving conditions, storage medium directory and file name format;

[0064] A file scanning module, used to scan all the data files to be archived in the specified directory;

[0065] A file mapping module is used to map each scanned data file to be archived into a corresponding data file, a check file and a ddl file, and add a corresponding data file record to be archived in a database table; the data file record to be archived includes the path information of the data file, the path information of the check file and the path information of the ddl file;

[0066] A database table reading module is used to read the database table in real time, and when a new data file record to be archived appears in the database table, the file archiving module is controlled to work;

[0067] An archiving information reading module is used to read the new data file to be archived and record the corresponding archiving information;

[0068] A data analysis module is used to compare the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived according to the archiving condition in the archiving information, and if they are the same, archive the data file corresponding to the new data file record to be archived and store it in the storage medium; if they are different, report an error and record the error in a log;

[0069] A data collection module is used to collect data files to be archived from data sources;

[0070] The data cache module is used to cache the collected data files to be archived into the message queue.

[0071] The first verification module is used to verify whether the check file is consistent before and after archiving, and verify whether the data file is consistent before and after archiving. If the check file or the data file is inconsistent before and after archiving, the archiving process is interrupted. If the check file and the data file are consistent before and after archiving, the ddl file is verified before and after archiving. If the newly added field is at the end, the archiving process continues. If the newly added field is not at the end, the archiving process is interrupted.

[0072] The second check module is used to create a temporary table, load the data in the data file into the temporary table, compare the number of data entries in the temporary table after the data is loaded with the number of data entries in the check, and record the data in the temporary table into the database table;

[0073] The log generation module is used to write the execution record into the log operation table of the database of the archiving system to generate an operation log.

[0074] A first monitoring module is used to monitor the archiving progress of data files; the archiving progress includes: archiving speed, archiving quantity, lost data files and duplicated archived data files;

[0075] A second monitoring module is used to monitor the storage medium status; the storage medium status includes: storage capacity, usage rate, hardware failure and network failure;

[0076] A fault handling module is used to re-archive or supplement the lost data files when data files are lost during the data file archiving process;

[0077] The report generation module is used to generate an archiving report; the archiving report includes archiving progress monitoring results, storage medium monitoring results and exception handling results; the exception handling results include: data file loss handling results, hardware failure handling results and network failure handling results.

[0078] A data backup module is used to perform archived data backup; the archived data backup includes: backup method, backup cycle, backup storage medium, backup data verification and backup recovery; the backup method includes: full backup, incremental backup or selected field backup; the backup storage medium includes: solid state drive; the backup data verification includes: readability verification and consistency verification; the backup recovery includes: providing data recovery when the stored data is damaged or the data file is unavailable.

[0079] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data file archiving method, characterized in that: The following steps are involved: Establish a streaming data processing platform in a data center or cloud platform; Configure archiving information for each data file to be archived in the specified directory; The archiving information includes: archiving target type, archiving life cycle, archiving conditions, file name and storage medium; archiving target type refers to the data classification type of the archiving target, including: main file object, zipper object and detail file object; archiving life cycle refers to the life cycle or storage duration of the archived data, including: daily storage, weekly storage, monthly storage, annual storage and permanent storage; archiving conditions refer to the conditions for executing archiving, including: daily execution, execution at the end of the month, execution at the end of ten days, execution at the end of the quarter and execution at the end of the year; file name means that the name of the archived object must comply with the naming specification, including: system name_table name; The following steps are performed on the streaming data processing platform: Scan all the data files to be archived under the specified directory; Map each scanned data file to be archived into a corresponding data file, a check file and a ddl file, and add a corresponding data file record to be archived in the database table; the data file record to be archived includes the path information of the data file, the path information of the check file and the path information of the ddl file; The database table is read in real time, and when a new data file record to be archived appears in the database table, steps S1 and S2 are executed; wherein, Step S1: Read the new data file to be archived and record the corresponding archiving information; Step S2: comparing the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived according to the archiving condition in the archiving information; if they are the same, archiving the data file corresponding to the new data file record to be archived and storing it in the storage medium; if they are different, reporting an error and recording the error in the log; After step S1, the following steps are also included: Verify whether the check file is consistent before and after archiving, and verify whether the data file is consistent before and after archiving. If the check file or the data file is inconsistent before and after archiving, the archiving process is interrupted. If the check file and the data file are consistent before and after archiving, verify whether the ddl file is consistent before and after archiving. If the newly added field is at the end, continue the archiving process. If the newly added field is not at the end, interrupt the archiving process. Create a temporary table, and load the data in the data file into the temporary table, compare the number of data entries in the temporary table after the data is loaded with the number of data entries in the check, and record the data in the temporary table into the database table; The execution record is written into the log operation table of the database of the archiving system to generate an operation log.

2. A data file archiving method according to claim 1, characterized in that: Before reading the database table in real time, the following steps are also included: Collect data files to be archived from data sources; Cache the collected data files to be archived into the message queue.

3. A data file archiving method according to claim 1 or 2, characterized in that: The data file archiving process also includes the following steps: Monitor the archiving progress of data files; the archiving progress includes: archiving speed, archiving quantity, lost data files and duplicated data files; Monitor the storage medium status; the storage medium status includes: storage capacity, usage rate, hardware failure and network failure; If data files are lost during the data file archiving process, re-archive or supplement the lost data files; Generate an archiving report; the archiving report includes archiving progress monitoring results, storage medium monitoring results and exception handling results; the exception handling results include: data file loss handling results, hardware failure handling results and network failure handling results.

4. A data file archiving method according to claim 1 or 2, characterized in that: After the data files are archived, the following steps are also included: backing up the archived data; the archived data backup includes: backup method, backup cycle, backup storage medium, backup data verification and backup recovery; the backup method includes: full backup, incremental backup or selected field backup; the backup storage medium includes: solid state drive; the backup data verification includes: readability verification and consistency verification; the backup recovery includes: providing data recovery when the stored data is damaged or the data file is unavailable.

5. A data file archiving system, characterized in that: include: A platform building module is used to build a streaming data processing platform on a data center or cloud platform; The information configuration module is used to configure archiving information for each data file to be archived under the specified directory; the archiving information includes: archiving target type, archiving life cycle, archiving conditions, file name and storage medium; the archiving target type refers to the data classification type of the archiving target, including: main file object, zipper object and detail file object; the archiving life cycle refers to the life cycle or storage duration of the archived data, including: daily storage, weekly storage, monthly storage, annual storage and permanent storage; the archiving conditions refer to the conditions for executing archiving, including: daily execution, execution at the end of the month, execution at the end of ten days, execution at the end of the quarter and execution at the end of the year; the file name means that the name of the archived object must comply with the naming specification, including: system name_table name; A file scanning module, used to scan all the data files to be archived in the specified directory; A file mapping module is used to map each scanned data file to be archived into a corresponding data file, a check file and a ddl file, and add a corresponding data file record to be archived in a database table; the data file record to be archived includes the path information of the data file, the path information of the check file and the path information of the ddl file; A database table reading module is used to read the database table in real time, and when a new data file record to be archived appears in the database table, the file archiving module is controlled to work; An archiving information reading module is used to read the new data file to be archived and record the corresponding archiving information; A data analysis module is used to compare the file name in the archiving information with the file name of the data file corresponding to the new data file record to be archived according to the archiving condition in the archiving information, and if they are the same, archive the data file corresponding to the new data file record to be archived and store it in the storage medium; if they are different, report an error and record the error in a log; Also includes: The first verification module is used to verify whether the check file is consistent before and after archiving, and verify whether the data file is consistent before and after archiving. If the check file or the data file is inconsistent before and after archiving, the archiving process is interrupted. If the check file and the data file are consistent before and after archiving, the ddl file is verified before and after archiving. If the newly added field is at the end, the archiving process continues. If the newly added field is not at the end, the archiving process is interrupted. The second check module is used to create a temporary table, load the data in the data file into the temporary table, compare the number of data entries in the temporary table after the data is loaded with the number of data entries in the check, and record the data in the temporary table into the database table; The log generation module is used to write the execution record into the log operation table of the database of the archiving system to generate an operation log.

6. A data file archiving system according to claim 5, characterized in that: Also includes: A data collection module is used to collect data files to be archived from data sources; The data cache module is used to cache the collected data files to be archived into the message queue.

7. A data file archiving system according to claim 5 or 6, characterized in that: Also includes: A first monitoring module, used to monitor the archiving progress of data files; The archiving progress includes: archiving speed, archiving quantity, lost data files and duplicated archived data files; A second monitoring module is used to monitor the storage medium status; the storage medium status includes: storage capacity, usage rate, hardware failure and network failure; A fault handling module is used to re-archive or supplement the lost data files when data files are lost during the data file archiving process; The report generation module is used to generate an archiving report; the archiving report includes archiving progress monitoring results, storage medium monitoring results and exception handling results; the exception handling results include: data file loss handling results, hardware failure handling results and network failure handling results.

8. A data file archiving system according to claim 5 or 6, characterized in that: Also includes: Data backup module, used for archiving data backup; The archived data backup includes: backup method, backup cycle, backup storage medium, backup data verification and backup recovery; the backup method includes: full backup, incremental backup or selected field backup; the backup storage medium includes: solid state drive; the backup data verification includes: readability verification and consistency verification; the backup recovery includes: providing data recovery when the stored data is damaged or the data file is unavailable.

Images