Interval type increment data reduction method, device and equipment and medium

Abstract

The embodiment of the invention discloses a data restoration method, device and equipment for interval increment, and a medium. The method comprises the steps that if it is detected that total data of the previous day is stored, incremental data of the day in a source business system database table is acquired; the intraday incremental data is obtained at the end time point of the intraday; and before the day end time point of the next day, performing data updating based on the stored total data of the previous day and the incremental data of the current day, and determining and storing the total data of the current day used by the batch processing system. Through the technical scheme of the embodiment of the invention, a large amount of time can be saved in the data extraction and data storage process by utilizing an interval type increased total amount staggered day storage mode, and the data timeliness and accuracy are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a method, device, equipment, and medium for restoring data with intermittent incremental amounts. Background Art

[0002] In the process of big data processing, it is often necessary to obtain accurate data information from the database tables of the source business system (referred to as the source system) at the end of the day and extract it into data files. The content of these data files is then widely used in important scenarios such as data analysis and mining, artificial intelligence, etc. Big data refers to large or complex data sets that are difficult for traditional data processing application software to handle.

[0003] The current challenge is how to solve the problem of not being able to guarantee that all the data in the large file is the same as the end of the day when extracting large tables from the database into big data files due to the real-time dynamic changes in the data within the database and the long time required for data extraction. For example, in the process of big data processing, it is necessary to obtain the accurate end of the day time point in a timely manner for data processing. However, when extracting data files, the dynamic changes in the database will cause the extracted data files to contain cross-day data. If the full amount of data in the data table is extracted, due to the long time consumption, the extraction period will be greatly affected by the dynamic changes in the database, and the extracted data will contain a large amount of cross-day data, and the cross-day data will have a wide range of influence. It can be seen that there is an urgent need for a way to ensure the timeliness and accuracy of big data extraction. Summary of the Invention

[0004] The embodiments of the present invention provide a data restoration method, device, equipment and medium for interval-based incremental data, which utilizes the staggered storage method of interval-based incremental data to save a lot of time in the data extraction and data storage process, thereby improving the timeliness and accuracy of the data.

[0005] In a first aspect, an embodiment of the present invention provides a data restoration method with intermittent incremental data, comprising:

[0006] If it is detected that the full data of the previous day is stored, the incremental data of the current day in the database table of the source business system is obtained; the incremental data of the current day is obtained at the end of the current day;

[0007] Before the end of the next day, data is updated based on the stored full data of the previous day and the incremental data of the current day, and the full data of the current day used by the batch processing system is determined and stored in the database.

[0008] Optionally, the method also includes: performing a data comparison between the stored full data of the previous day and the incremental data of the current day based on a preset data identifier, and determining a data comparison result corresponding to each preset data identifier; the preset data identifier is a data identifier corresponding to the data required by the batch processing system; updating the full data of the previous day based on each of the data comparison results and the incremental data of the current day, and determining the full data of the current day for use by the batch processing system and storing it in the database.

[0009] Optionally, the method also includes: for each preset data identifier, if the data comparison result of the current preset data identifier is that the data is different, then the full data of the previous day corresponding to the current preset data identifier is modified based on the incremental data of the day corresponding to the current preset data identifier, and the full data of the day for use by the batch processing system is determined and stored in the database.

[0010] Optionally, the method further includes: the incremental data for the day includes: the last modification date of the field, the second last modification date of the field, the last modified field and the second last modified field.

[0011] Optionally, the method also includes: modifying the previous day's full data corresponding to the current preset data identifier based on the current day's incremental data corresponding to the current preset data identifier, and determining the full data to be selected after the data update; comparing the last modification date of the field corresponding to each last modified field in the full data to be selected with the current day's date, and determining the date comparison result corresponding to each last modified field in the full data to be selected; based on the date comparison result, the second last modification date of the field and the second last modified field in the current day's incremental data, modifying the last modified field and the last modification date of the field in the full data to be selected, and determining the full data of the day used by the batch processing system and warehousing it.

[0012] Optionally, the method further includes: if it is detected that the full data of the previous day is not stored, obtaining the incremental data of the day and the full data of the previous day in the source business system database table.

[0013] In a second aspect, an embodiment of the present invention further provides a data restoration device with intermittent incremental data, the device comprising:

[0014] The daily incremental data acquisition module is used to acquire the daily incremental data from the source business system database table if it is detected that the full data of the previous day is stored; the daily incremental data is acquired at the end of the day;

[0015] The module for determining the full amount of data for the day is used to update the data based on the stored full amount of data of the previous day and the incremental data of the day before the end of the next day, determine the full amount of data of the day for use by the batch processing system and store it in the database.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0017] one or more processors;

[0018] a memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the data restoration method with interval increment as provided in any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data restoration method with interval-incremental amounts as provided in any embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the data restoration method with interval increments as provided in any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention is to obtain the incremental data of the day in the database table of the source business system if it is detected that the full data of the previous day is stored; the incremental data of the day is obtained at the end of the day; before the end of the next day, data is updated based on the stored full data of the previous day and the incremental data of the day, and the full data of the day used by the batch processing system is determined and stored in the warehouse, thereby utilizing the intermittent incremental full data staggered storage method, which can save a lot of time in the process of data extraction and data storage, and improve the timeliness and accuracy of the data.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flow chart of a data restoration method with interval increment provided by the first embodiment of the present invention;

[0026] Figure 2This is a flow chart of a data restoration method with intermittent incremental data provided by the second embodiment of the present invention;

[0027] Figure 3 This is a schematic structural diagram of a data restoration device with intermittent incremental data provided by the third embodiment of the present invention;

[0028] Figure 4 It is a structural diagram of an electronic device for implementing the data restoration method of the intermittent incremental amount according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Example 1

[0032] Figure 1 A flowchart of a data restoration method for interval-incremental data is provided for the first embodiment of the present invention. This embodiment is applicable to the case of extracting large data. The method can be executed by a data restoration device for interval-incremental data. The data restoration device for interval-incremental data can be implemented in the form of hardware and / or software. The data restoration device for interval-incremental data can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110. If it is detected that the full data of the previous day is stored, the incremental data of the current day in the source business system database table is obtained; the incremental data of the current day is obtained at the end of the current day.

[0034] Among them, the full data may refer to all the data in the database table of the source business system. The full data of the previous day may refer to the full data required by the batch processing system to be updated up to the end of the previous day. The source business system may refer to the business system that actually generates real business data, referred to as the source system. The source business system can realize data extraction, that is, the source business system can export the data content in the database table as a data file that can be further used by other systems (such as batch processing systems). Incremental data may refer to data that has changed in the database table of the source business system compared to the last time. The incremental data of the day may refer to data in the database table of the source business system required by the batch processing system compared to the value of the full data of the previous day. The end of the day may refer to 24:00 on a natural day or the end time of the day under the time scale specified in a certain system. In this embodiment, the two are approximately the same.

[0035] In this embodiment, the source business system submits data acquisition instructions based on the downstream system, such as the batch processing system, to determine the data required by each downstream system, and begins to extract the daily incremental data of the required data at the end of the day, and sends the extracted daily incremental data to the downstream system, or stores it in the data temporary storage of the corresponding downstream system so that the downstream system can automatically obtain it.

[0036] Specifically, if the batch processing system located downstream of the source business system has full data from the previous day, indicating that the incremental data has a basis for data modification, then the incremental data for the day in the database table of the source business system is obtained.

[0037] S120. Before the end of the next day, update the data based on the stored full data of the previous day and the incremental data of the current day, determine the full data of the current day for use by the batch processing system, and store it in the database.

[0038] The term "full daily data" refers to the full data required by the batch processing system to be updated up to the end of the day. The batch processing system can be used for batch data processing. Specifically, after receiving data files from upstream data systems, the batch processing system processes the data according to specific business rules. Interval-based full data increments refer to the difference between the data dates of the full data and the incremental data. In this embodiment, this difference is 1.

[0039] Specifically, before the end of the next day, the data with numerical changes in the stored full data of the previous day is updated using the incremental data of the day, and the updated full data of the day that can be used by the batch processing system is obtained and stored in the warehouse.

[0040] In this embodiment, the current day is T, the previous day is T-1, and the next day is T+1. By using the staggered storage method of intermittent full data addition, it is only necessary to prioritize the storage of the T-day incremental data on T+1, and to compare and update the data with the T-1 full data. Assuming that the time for extracting incremental data is W and the time for extracting full data is V, in the process of extracting big data, V>>W, compared with the method of obtaining full data every day, a lot of time can be saved in the process of data extraction and data update storage, solving the problem of low timeliness.

[0041] On the basis of the above technical solution, "updating data based on the stored full data of the previous day and the incremental data of the current day, determining the full data of the current day for use by the batch processing system and merging it into the database" may include: comparing the stored full data of the previous day with the incremental data of the current day based on preset data identifiers, and determining the data comparison results corresponding to each preset data identifier; the preset data identifier is the data identifier corresponding to the data required by the batch processing system; updating the full data of the previous day based on each data comparison result and the incremental data of the current day, and determining the full data of the current day for use by the batch processing system and merging it into the database.

[0042] The preset data identifier may refer to a unique identifier set by the source business system when the data is generated. For example, the preset data identifier may be, but is not limited to, an ID such as 001. The data comparison result may refer to the comparison result of the data numerical consistency.

[0043] Specifically, based on the preset data identifier, the stored full data of the previous day and the incremental data of the current day are compared to determine the data comparison result corresponding to each preset data identifier; if the data comparison result is that the data are the same, there is no need to modify the data value corresponding to the data comparison result, and the modification time of the data value can be updated to achieve accurate data update, and the full data of the current day for use by the batch processing system is obtained and stored in the database.

[0044] In this embodiment, the current day's incremental data is stored in the first table (K table) according to the current day's date, and the previous day's full data is copied to the second table (O' table) according to the previous day's date. The data date in the second table is updated to the current day (T). The O' table and the K table are linked based on the primary key (preset data identifier) ​​in the database table, and the previous day's full data is compared with the current day's incremental data. Data that can be successfully linked and has completely consistent values ​​is not modified.

[0045] On the basis of the above technical solution, "updating the full data of the previous day based on the comparison results of each data and the incremental data of the day, determining the full data of the day for use by the batch processing system and merging it into the database" may include: for each preset data identifier, if the data comparison result of the current preset data identifier is that the data is different, then modifying the full data of the previous day corresponding to the current preset data identifier based on the incremental data of the day corresponding to the current preset data identifier, determining the full data of the day for use by the batch processing system and merging it into the database.

[0046] Among them, there are two data update methods for data comparison results with different data.

[0047] In this embodiment, the O' table and the K table are linked based on the primary key (preset data identifier) ​​in the database table, and the full data of the previous day is compared with the incremental data of the current day. On this basis, for data that can be successfully linked but the values ​​are not completely consistent, the data corresponding to the same primary key in the O' table is updated based on the incremental data of the current day. For data that only exists in the K table, such as new data, it can be directly inserted into the O' table.

[0048] The technical solution of the embodiment of the present invention is to obtain the incremental data of the day in the database table of the source business system if it is detected that the full data of the previous day is stored; the incremental data of the day is obtained at the end of the day; before the end of the next day, the data is updated based on the stored full data of the previous day and the incremental data of the day, and the full data of the day used by the batch processing system is determined and stored in the warehouse, thereby utilizing the intermittent incremental full data staggered storage method, which can save a lot of time in the process of data extraction and data storage, and improve the timeliness and accuracy of the data.

[0049] Based on the above technical solution, the method also includes: if it is detected that the full data of the previous day is not stored, obtaining the incremental data of the day and the full data of the previous day in the source business system database table.

[0050] Specifically, if it is detected that the full data of the previous day is not stored, indicating that the downstream system is requesting data from the source business system for the first time, the incremental data of the day and the full data of the previous day in the database table of the source business system are obtained, and the full data of the previous day is used as the data update basis for the incremental data of the day.

[0051] In this embodiment, if the full data of the previous day is empty, the incremental data of the current day is used as the full data of the current day and stored in the database.

[0052] Example 2

[0053] Figure 2This is a flowchart of a data restoration method for interval-based full-data provided by the second embodiment of the present invention. Based on the above embodiment, this embodiment describes in detail the process of determining the full data of the day. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0054] S210. If it is detected that the full data of the previous day is stored, the incremental data of the current day in the source business system database table is obtained; the incremental data of the current day is obtained at the end of the current day.

[0055] S220. Before the end of the next day, compare the stored full data of the previous day with the incremental data of the current day based on preset data identifiers to determine the data comparison results corresponding to each preset data identifier; the preset data identifiers are data identifiers corresponding to the data required by the batch processing system.

[0056] S230. For each preset data identifier, if the data comparison result of the current preset data identifier is that the data is different, the full data of the previous day corresponding to the current preset data identifier is modified based on the incremental data of the day corresponding to the current preset data identifier, and the full data to be selected after the data update is determined.

[0057] Among them, the incremental data of the day includes: the last modification date of the field, the second last modification date of the field, the field after the last modification, and the field after the second last modification. The full data to be selected may refer to the full data of the day after the preliminary update. There may be cross-day data in the full data to be selected. Cross-day data may refer to the data content extracted after the end of the day and the data that has changed again in the early morning of the next day due to the dynamic characteristics of the database and the continuity of data extraction. For example, when extracting data from a data table, due to the long time consumption and the large impact of the dynamic change characteristics of the database during the extraction period, there are more cross-day data in the extracted data, and the cross-day data has a large impact range, which cannot meet the application scenarios with high timeliness and high data accuracy requirements. Dynamic changes may refer to the operation in which the content of the data table in the database is modified at a certain moment or within a certain time period due to 24-hour business.

[0058] In this embodiment, the data table structure for extracting data from the source business system is optimized. The data table structure for extracting data from the source business system can be optimized by optimizing the source system database table structure. It is also possible to optimize the data extraction request sent to the source business system, such as requiring data to be extracted and issued according to a specific data table structure. The last modified date of the field, the second last modified date of the field, the last modified field, and the second last modified field can be understood as key focus fields. Based on the key focus fields, other data under the same primary key can also be extracted. The key focus fields are optimized from the original single field to a group of fields, specifically the four fields of the last modified date of the field, the second last modified date of the field, the last modified field, and the second last modified field. Assume that for a database table recording customer information, incremental data from 20240131 (T day) is extracted in the early morning of 20240201 (T+1 day) for data processing in the batch processing system. After the end of the day on 20240131 (T day) and before the data extraction is completed, the source business system modifies the customer name (field). The recording process before and after the database table structure optimization is shown in Table 1. Obviously, the optimized group field records the change process, and the data before the end of the day can be restored through the change process.

[0059] Table 1 Example of table content changes before and after optimization

[0060]

[0061] Among them, before the table structure is optimized, the customer name field only has the customer name. Moreover, the data value changes during the data extraction process, which will lead to inaccurate incremental data acquisition. "Zhang Xiaoming" should be the accurate data content at the end of the day, but because of the data extraction process, that is, after the end of the day, "Zhang Xiaoming" was changed to "Zhang Daming", resulting in the actual extracted data being "Zhang Daming", which is inaccurate. In this embodiment, the data extracted after the table structure is optimized may include: the last modification date of the field, the second last modification date of the field, the last modified field, and the second last modified field. These key fields can be used to accurately distinguish the specific modification time of the data, so that data can be restored later. Data restoration can refer to the process of correcting data content that does not meet expectations to the required accurate data by focusing on fields.

[0062] S240 . Determine a date comparison result corresponding to each last modified field in the full amount of data to be selected based on a comparison between the last modified date of the field corresponding to each last modified field in the full amount of data to be selected and the current date.

[0063] The date comparison result may refer to a comparison result of date consistency, and may include: date consistency and date inconsistency.

[0064] S250. Based on the date comparison result, the last modified date of the field in the incremental data of the day, and the last modified field of the day, modify the last modified field and the last modified date of the field in the selected full data, determine the full data of the day used by the batch processing system, and store it in the database.

[0065] Specifically, if the date comparison result indicates a match, the data value remains unchanged. If the date comparison result indicates a match, the last modified field in the selected full data set is changed to the next last modified field, and the last modified date in the selected full data set is changed to the next last modified date. For example, in Table 1, "Zhang Daming" is changed to "Zhang Xiaoming," and the modification date is changed from "20240201" to "20230513."

[0066] In this embodiment, if the full data of day T is processed on day T+1, and there is a case in the full data to be selected where the last modification date of a field is greater than day T (the date of the day), then the last modification date of the field in the O' table is restored to the modification date of the field on the next last day, and the last modified field is restored to the modified field on the next last day. After restoration, the full data of day T in the O' table is the accurate full data at the end of day T, which can be used in scenarios with high timeliness and high accuracy requirements. The technical solution of the embodiment of the present invention can accurately restore the data of the key fields at the end of day T by adding data table structure fields, solving the pain point of inaccurate cross-day data in traditional solutions.

[0067] It should be noted that the batch processing system staggers the storage of the full amount of data on day T exported from the source system before the end of day T+1, and stores it in table O according to the date, to be used when processing the full amount of data on day T+1 on day T+2. In this embodiment, the storage of the full amount of data on day T only needs to be completed before the end of day T+1, and the full amount of data on day T-1 has been completed before the end of day T and stored in table O according to the date. Therefore, the technical solution of this embodiment can avoid data errors caused by the long-term accumulation of incremental data, control data inconsistency errors within one day, and eliminate data errors existing on the previous day on the next day, solving the problem of low data accuracy in the traditional data zipper table.

[0068] The technical solution of the embodiment of the present invention is to determine the full data to be selected after the data update by modifying the full data of the previous day corresponding to the current preset data identifier based on the current day incremental data corresponding to the current preset data identifier; determine the date comparison result corresponding to each last modified field in the full data to be selected based on the comparison of the field last modification date and the current day date corresponding to each last modified field in the full data to be selected; modify the last modified field and the field last modification date in the full data to be selected based on the date comparison result, the field second last modification date and the field modified on the second last day in the incremental data of the current day, determine the full data of the current day used by the batch processing system and put it into the database, thereby determining and restoring the cross-day data by adding auxiliary fields of the key focus fields to the source system database table to avoid the occurrence of cross-day data, further improving the timeliness and accuracy of full data acquisition, and thus making it possible to determine the full data of the current day to meet the data application scenarios with high timeliness and high accuracy.

[0069] The following is an embodiment of a data restoration device for intermittently increasing the amount of data provided by an embodiment of the present invention. The device and the data restoration methods for intermittently increasing the amount of data in the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the data restoration device for intermittently increasing the amount of data, please refer to the embodiment of the data restoration method for intermittently increasing the amount of data in the above-mentioned embodiments.

[0070] Example 3

[0071] Figure 3 This is a schematic diagram of the structure of a data restoration device with intermittent increment provided by the third embodiment of the present invention. Figure 3 As shown, the device includes: a daily incremental data acquisition module 310 and a daily full data determination module 320.

[0072] Among them, the daily incremental data acquisition module 310 is used to obtain the daily incremental data in the source business system database table if it is detected that the full data of the previous day is stored; the daily incremental data is obtained at the end of the day; the daily full data determination module 320 is used to update the data based on the stored full data of the previous day and the daily incremental data before the end of the next day, determine the daily full data for use by the batch processing system and store it in the warehouse.

[0073] The technical solution of the embodiment of the present invention is to obtain the incremental data of the day in the database table of the source business system if it is detected that the full data of the previous day is stored; the incremental data of the day is obtained at the end of the day; before the end of the next day, the data is updated based on the stored full data of the previous day and the incremental data of the day, and the full data of the day used by the batch processing system is determined and stored in the warehouse, thereby utilizing the intermittent incremental full data staggered storage method, which can save a lot of time in the process of data extraction and data storage, and improve the timeliness and accuracy of the data.

[0074] Based on the above technical solution, the daily full data determination module 320 may include:

[0075] The data comparison result determination submodule is used to compare the stored full data of the previous day with the incremental data of the current day based on preset data identifiers, and determine the data comparison result corresponding to each preset data identifier; the preset data identifier is the data identifier corresponding to the data required by the batch processing system;

[0076] The submodule for determining the full amount of data for the day is used to update the full amount of data for the previous day based on the comparison results of each data and the incremental data of the day, determine the full amount of data for the day used by the batch processing system and store it in the database.

[0077] Based on the above technical solution, the submodule for determining the full amount of data for the day may include:

[0078] The unit for determining the full amount of data for the day is used to modify the full amount of data of the previous day corresponding to the current preset data identifier based on the incremental data of the day corresponding to the current preset data identifier for each preset data identifier, if the data comparison result of the current preset data identifier is that the data is different, and determine the full amount of data of the day for use by the batch processing system and store it in the database.

[0079] Based on the above technical solution, the incremental data of the day includes: the last modification date of the field, the second last modification date of the field, the last modified field and the second last modified field.

[0080] On the basis of the above technical solution, the unit for determining the full data of the day is specifically used to: modify the full data of the previous day corresponding to the current preset data identifier based on the incremental data of the day corresponding to the current preset data identifier, and determine the full data to be selected after the data is updated; compare the last modification date of the field corresponding to each last modified field in the full data to be selected with the date of the day, and determine the date comparison result corresponding to each last modified field in the full data to be selected; based on the date comparison result, the second last modification date of the field in the incremental data of the day, and the second last modified field, modify the last modified field and the last modification date of the field in the full data to be selected, determine the full data of the day used by the batch processing system and put it into the warehouse.

[0081] On the basis of the above technical solution, the device further includes:

[0082] The incremental full data acquisition module is used to obtain the incremental data of the current day and the full data of the previous day from the source business system database table if it is detected that the full data of the previous day is not stored.

[0083] The data restoration device for intermittently increasing the amount provided by the embodiment of the present invention can execute the data restoration method for intermittently increasing the amount provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the data restoration method for intermittently increasing the amount.

[0084] It is worth noting that in the above-mentioned embodiment of interval-based incremental data restoration, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0085] Example 4

[0086] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0087] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0088] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0089] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the interval-based incremental data restoration method.

[0090] In some embodiments, the data restoration method for interval increments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data restoration method for interval increments described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data restoration method for interval increments by any other appropriate means (e.g., by means of firmware).

[0091] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0093] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0095] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0096] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0097] An embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the data restoration method with interval increments as provided in any embodiment of the present application.

[0098] During the implementation of the computer program product, the computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). This program product and the interval-incremented data restoration method disclosed in each embodiment of the present application belong to the same inventive concept, so they are not described here.

[0099] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0100] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data restoration method with interval increment, characterized in that: include: If it is detected that the full data of the previous day is stored, the incremental data of the day in the source business system database table is obtained; The daily incremental data is obtained at the end of the day; Before the end of the next day, data is updated based on the stored full data of the previous day and the incremental data of the current day, and the full data of the current day used by the batch processing system is determined and stored in the database.

2. The method according to claim 1, characterized in that The data updating based on the stored full data of the previous day and the incremental data of the current day, determining the full data of the current day for use by the batch processing system and storing it in the database, includes: Based on the preset data identifiers, the stored full data of the previous day is compared with the incremental data of the current day, and the data comparison result corresponding to each preset data identifier is determined; the preset data identifier is the data identifier corresponding to the data required by the batch processing system; Based on the data comparison results and the incremental data of the day, the full data of the previous day is updated, and the full data of the day for use by the batch processing system is determined and stored in the database.

3. The method according to claim 2, characterized in that The updating of the previous day's full data based on the data comparison results and the current day's incremental data, and determining the current day's full data for use by the batch processing system and storing the data in the database, includes: For each preset data identifier, if the data comparison result of the current preset data identifier is that the data is different, the full data of the previous day corresponding to the current preset data identifier is modified based on the incremental data of the day corresponding to the current preset data identifier, and the full data of the day used by the batch processing system is determined and stored in the database.

4. The method according to claim 3, characterized in that The incremental data for the day includes: the last modified date of the field, the second last modified date of the field, the last modified field, and the second last modified field.

5. The method according to claim 4, characterized in that Modifying the previous day's full data corresponding to the current preset data identifier based on the current day's incremental data corresponding to the current preset data identifier, determining the current day's full data for use by the batch processing system and storing it in the database, including: Modify the previous day's full data corresponding to the current preset data identifier based on the current day's incremental data corresponding to the current preset data identifier, and determine the updated full data to be selected; Determine a comparison result of the date corresponding to each last modified field in the full amount of data to be selected based on a comparison of the last modified date of the field corresponding to each last modified field in the full amount of data to be selected with the current date; Based on the date comparison result, the last modified date of the field in the incremental data of the day and the modified field on the second last day, the last modified field and the last modified date of the field in the full data to be selected are modified to determine the full data of the day used by the batch processing system and put it into the database.

6. The method according to claim 1, characterized in that The method further comprises: If it is detected that the full data of the previous day is not stored, the incremental data of the day and the full data of the previous day in the database table of the source business system are obtained.

7. A data restoration device with intermittent incremental data, characterized in that: The device comprises: The daily incremental data acquisition module is used to acquire the daily incremental data from the source business system database table if it is detected that the full data of the previous day is stored; the daily incremental data is acquired at the end of the day; The module for determining the full amount of data for the day is used to update the data based on the stored full amount of data of the previous day and the incremental data of the day before the end of the next day, determine the full amount of data of the day for use by the batch processing system and store it in the database.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data restoration method with interval increment as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for restoring data with interval increment as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data restoration method with interval increment as described in any one of claims 1 to 6.

Images