Data archiving method, apparatus, computer device and storage medium

Abstract

The application relates to the technical field of big data processing, in particular to a data archiving method and device, computer equipment, a storage medium and a computer program product. The archiving start parameters and the archiving execution parameters corresponding to a target data warehouse are acquired. When it is determined according to the archiving start parameters that the target data warehouse needs to perform data archiving, the to-be-archived zipped list in the target data warehouse is determined based on the archiving execution parameters. Then, the zipped data in the to-be-archived zipped list that meets the preset open-close chain rule is subjected to data open-close chain processing to obtain a target zipped list. The data archiving processing can be directly performed on the to-be-archived zipped list that may contain un-closed zipped data, so that the slice data of some time points of the data object is prevented from being lost. Finally, the target zipped list is subjected to data archiving and cleaning processing based on the archiving execution parameters, the data archiving and open-close chain actions are automatically executed, the work intensity and the failure rate of operation and maintenance personnel are reduced, and automatic and effective data archiving is realized.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data archiving method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of Internet and big data technologies, the amount of business data that needs to be stored and processed in various industries is growing exponentially. In order to effectively cope with the massive amount of business data, data warehouses based on distributed systems are usually used to replace traditional databases to store and manage the business data.

[0003] To effectively address the issue of rapidly declining data warehouse operation and query performance caused by the accumulation of large amounts of data after prolonged operation, existing enterprises are currently undertaking data warehouse archiving and upgrades. Current data archiving methods, due to the massive amount of data to be maintained and the complex processes involved, typically require extensive training of maintenance personnel before execution. In practice, maintenance personnel need to input data or perform operations multiple times, and then the computer processes and responds to the input data and operations to perform data archiving. However, because the computer can only passively respond and process in this process, errors in the data input or operations performed by the operators will directly lead to data archiving failure, thus failing to achieve automatic and effective data archiving. Summary of the Invention

[0004] Therefore, it is necessary to provide an automatic and effective data archiving method, apparatus, computer equipment, storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a data archiving method. The method includes:

[0006] Obtain the data archiving target parameters corresponding to the target data warehouse; the data archiving target parameters include archiving start parameters and archiving execution parameters.

[0007] If it is determined from the archive startup parameters that the target data warehouse needs to be archived, the archive execution parameters are used to determine the zipper table to be archived in the target data warehouse.

[0008] Perform data opening and closing chaining processing on the zipper data in the zipper table to be archived that conform to the preset opening and closing chaining rules to obtain the target zipper table;

[0009] Based on the archive execution parameters, perform data archiving and cleanup processing on the target zipper table.

[0010] In one embodiment, the archiving start parameters include a preset archiving period and the current number of tasks in the target data warehouse;

[0011] Determining whether the target data warehouse needs data archiving based on the archiving startup parameters includes:

[0012] If the time interval between the current system date and the previous archive date reaches the preset archive period, and the current number of tasks in the target data warehouse is zero, then it is determined that the target data warehouse needs to perform data archiving.

[0013] In one embodiment, performing data opening and closing chaining processing on the zipper data in the zipper table to be archived that conforms to preset opening and closing chaining rules includes:

[0014] For zipper data in the pending archive zipper table whose opening date is earlier than the current system date and which does not have a closing date, perform data re-closing processing;

[0015] Data truncation is performed on zipper data in the zipper table to be archived where the opening date is less than the current system date and the closing date is greater than the current system date.

[0016] In one embodiment, the step of performing data re-closing processing on zipper data in the zipper table to be archived whose opening date is less than the current system date and which does not have a closing date includes:

[0017] Set the closing date of the zipper data whose opening date is less than the current system date and which has no closing date to the current system date;

[0018] Based on the zipper data whose opening date is less than the current system date and has no closing date, add zipper data whose opening date is the current system date and whose closing date is a preset maximum date.

[0019] In one embodiment, the data archiving and cleaning process performed on the target zipper table based on the archiving execution parameters includes:

[0020] Based on the archive execution parameters, perform data archiving processing on the zipper data in the target zipper table whose opening date is less than the current system date;

[0021] After the data archiving process is completed, data cleaning is performed on the zipper data in the target zipper table whose opening date is earlier than the current system date.

[0022] In one embodiment, the archiving execution parameters include the split archiving method and the nearline storage cycle;

[0023] The data archiving process for zipper data in the target zipper table whose opening date is less than the current system date, based on the archiving execution parameters, includes:

[0024] According to the splitting and archiving method, zipper data in the target zipper table whose time interval from the opening date to the current system date is less than or equal to the near-line storage period is stored near-line, and zipper data in the target zipper table whose time interval from the opening date to the current system date is greater than the near-line storage period is stored offline.

[0025] In one embodiment, after obtaining the data archiving target parameters corresponding to the target data warehouse, the method further includes:

[0026] Generate an archiving script corresponding to the data archiving target parameters. The archiving script includes a main program script, a data warehouse execution script, and a data archiving script.

[0027] The step of determining the list of data to be archived in the target data warehouse based on the archive execution parameters when it is determined that the target data warehouse needs to be archived according to the archive startup parameters includes: determining the list of data to be archived in the target data warehouse through the main program script and based on the archive execution parameters when it is determined that the target data warehouse needs to be archived according to the archive startup parameters;

[0028] The step of performing data opening and closing chaining processing on the zipper data in the zipper table to be archived that conforms to the preset opening and closing chaining rules to obtain the target zipper table includes: performing data opening and closing chaining processing on the zipper data in the zipper table to be archived that conforms to the preset opening and closing chaining rules through the data warehouse execution script to obtain the target zipper table;

[0029] The process of performing data archiving and cleaning on the target zipper table based on the archiving execution parameters includes: performing data archiving and cleaning on the target zipper table through the data archiving script, the data warehouse execution script, and based on the archiving execution parameters.

[0030] In one embodiment, the method further includes:

[0031] Obtain the execution log file corresponding to the archived script;

[0032] The execution log file is analyzed and inspected.

[0033] Secondly, this application also provides a data archiving apparatus. The apparatus includes:

[0034] The parameter acquisition module is used to acquire the data archiving target parameters corresponding to the target data warehouse; the data archiving target parameters include archiving start parameters and archiving execution parameters.

[0035] The zipper table determination module is used to determine the zipper table to be archived in the target data warehouse based on the archive execution parameters when it is determined from the archive startup parameters that the target data warehouse needs to be archived.

[0036] The open / closed chain processing module is used to perform data open / closed chain processing on the zipper data in the zipper table to be archived that conforms to the preset open / closed chain rules, so as to obtain the target zipper table.

[0037] The archive cleanup module is used to perform data archiving and cleanup processing on the target zipper table based on the archive execution parameters.

[0038] In one embodiment, the archiving start parameters include a preset archiving period and the current number of tasks in the target data warehouse;

[0039] The device further includes:

[0040] The archiving judgment module is used to determine that the target data warehouse needs to be archived when the time interval between the current system date and the previous archiving date reaches the preset archiving period and the current number of tasks in the target data warehouse is zero.

[0041] In one embodiment, the open / closed chain processing module includes:

[0042] The re-closing chain processing module is used to perform data re-closing chain processing on the zipper data in the zipper table to be archived where the opening date is earlier than the current system date and there is no closing date.

[0043] The truncation processing module is used to perform data truncation processing on zipper data in the zipper table to be archived where the opening date is less than the current system date and the closing date is greater than the current system date.

[0044] In one embodiment, the re-closing chain processing module is further configured to set the closing date of zipper data whose opening date is less than the current system date and which does not have a closing date to the current system date; and to add zipper data whose opening date is less than the current system date and which does not have a closing date based on the zipper data whose opening date is less than the current system date and which does not have a closing date, with the opening date being the current system date and the closing date being a preset maximum date.

[0045] In one embodiment, the archive cleanup module includes:

[0046] The data archiving module is used to perform data archiving processing on zipper data in the target zipper table whose opening date is less than the current system date, based on the archiving execution parameters;

[0047] The data cleaning module is used to perform data cleaning processing on the zipper data in the target zipper table whose opening date is less than the current system date after the data archiving process is completed.

[0048] In one embodiment, the archiving execution parameters include the split archiving method and the nearline storage cycle;

[0049] The data archiving module is further configured to perform near-line storage on zipper data in the target zipper table whose time interval between the opening date and the current system date is less than or equal to the near-line storage period, and to perform offline storage on zipper data in the target zipper table whose time interval between the opening date and the current system date is greater than the near-line storage period, according to the split archiving method.

[0050] In one embodiment, the device further includes:

[0051] The script generation module is used to generate the archiving script corresponding to the data archiving target parameters. The archiving script includes a main program script, a data warehouse execution script, and a data archiving script.

[0052] The zipper table determination module is further configured to determine the zipper table to be archived in the target data warehouse by means of the main program script and based on the archive execution parameters when it is determined that the target data warehouse needs to be archived according to the archive startup parameters.

[0053] The open / closed chain processing module is also used to perform data open / closed chain processing on the zipper data in the zipper table to be archived that conforms to the preset open / closed chain rules through the data warehouse execution script, so as to obtain the target zipper table;

[0054] The archive cleanup module is also used to perform data archiving and cleanup processing on the target zipper table using the data archiving script, the data warehouse execution script, and based on the archive execution parameters.

[0055] In one embodiment, the device further includes:

[0056] The log acquisition module is used to acquire the execution log file corresponding to the archived script;

[0057] The log checking module is used to analyze and check the execution log file.

[0058] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0059] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0060] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0061] The aforementioned data archiving methods, devices, computer equipment, storage media, and computer program products acquire data archiving target parameters corresponding to the target data warehouse. These parameters include archiving initiation parameters and archiving execution parameters. If the target data warehouse requires data archiving based on the archiving initiation parameters, the archived zipper table in the target data warehouse is determined based on the archiving execution parameters. Then, data opening and closing processing is performed on the zipper data in the archived zipper table that conforms to preset opening and closing rules to obtain the target zipper table. This avoids directly archiving the archived zipper table, which may contain unclosed zipper data, thus preventing the loss of slice data at certain points in time. Finally, data archiving and cleanup processing is performed on the target zipper table based on the archiving execution parameters. This automated data archiving and opening / closing actions significantly reduce the workload and error rate of maintenance personnel through a simple operation process, achieving automatic and effective data archiving. Attached Figure Description

[0062] Figure 1 This is a diagram illustrating the application environment of a data archiving method in one embodiment;

[0063] Figure 2 This is a flowchart illustrating a data archiving method in one embodiment;

[0064] Figure 3 This is a flowchart illustrating a data archiving method in another embodiment;

[0065] Figure 4 This is a flowchart illustrating the data archiving method in yet another embodiment;

[0066] Figure 5 This is a flowchart illustrating the data archiving method in yet another embodiment;

[0067] Figure 6 This is a flowchart illustrating the data archiving method in yet another embodiment;

[0068] Figure 7 This is a structural block diagram of a data archiving device in one embodiment;

[0069] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0070] It should be noted that the user information (including but not limited to user device information, user personal information, etc. stored in the target data warehouse) and data (including but not limited to data used for analysis, stored data, and displayed data in the target data warehouse) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] The data archiving method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Specifically, server 104 obtains the data archiving target parameters corresponding to the target data warehouse sent by terminal 102; the data archiving target parameters include archiving start parameters and archiving execution parameters; if it is determined that the target data warehouse needs data archiving based on the archiving start parameters, the server determines the list of data to be archived in the target data warehouse based on the archiving execution parameters; the server performs data opening and closing chain logic processing on the data in the list of data to be archived that conforms to preset opening and closing chain rules to obtain the target list; and performs data archiving and cleaning logic processing on the target list based on the archiving execution parameters. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0073] In one embodiment, such as Figure 2 As shown, a data archiving method is provided, which can be applied to... Figure 1Taking server 104 as an example, the following S200 to S800 are included:

[0074] S200: Obtain the data archiving target parameters corresponding to the target data warehouse; the data archiving target parameters include archiving start parameters and archiving execution parameters.

[0075] The target data warehouse is a data warehouse that requires data archiving, typically built on a Hadoop distributed system architecture. The data warehouse involved in this application differs from a typical relational database; its data is stored in the form of a linked list to record historical information about changes to data objects. Each linked entry in the list includes both open and closed states, and corresponding open and close dates. Changes to the data object at any point in time, from its open date to the current time, can be retrieved through the linked list.

[0076] Specifically, data archiving target parameters represent the relevant parameters required for data archiving processing of the target data warehouse. These parameters can be obtained through parameter configuration operations performed by maintenance personnel on the terminal interface, or they can be obtained based on the operational status data of the target data warehouse. Data archiving target parameters may include archiving start parameters and archiving execution parameters. Archiving start parameters are used to determine whether the target data warehouse needs to be started for data archiving processing. Archiving execution parameters are used to determine the list of items in the target data warehouse that require data archiving processing and the method of data archiving processing.

[0077] Correspondingly, the types of parameters included in the archiving startup parameters are not unique and can be determined based on the parameter configuration of the maintenance personnel. This determines whether the target data warehouse needs to be started for data archiving. For example, archiving startup parameters may include a preset archiving date (e.g., December 31st), in which case the determination of whether the target data warehouse needs to be started for data archiving can be based on whether the current system date has reached the preset archiving date. Archiving startup parameters may also include a preset archiving period (weekly, monthly, or annually) or a preset archiving frequency (once a week, once a month, or once a year), in which case the determination of whether the target data warehouse needs to be started for data archiving can be based on whether the time interval between the current system date and the previous archiving date has reached the preset archiving period or preset archiving frequency. Archiving startup parameters may also include a preset upper limit data volume, in which case the determination of whether the target data warehouse needs to be started for data archiving can be based on whether the current stored data volume of the target data warehouse has reached the preset upper limit data volume. In addition, to avoid affecting the tasks being executed in the target data warehouse when starting data archiving, the archiving start parameters can also include the current number of tasks in the target data warehouse. This means that the target data warehouse can be started for data archiving based on whether the current number of tasks in the target data warehouse is zero.

[0078] It is understood that the archiving startup parameters can be any combination of the above parameters, which can be freely configured by maintenance personnel on the terminal page as needed. For example, in one embodiment, the archiving startup parameters include a preset archiving period and the current number of tasks in the target data warehouse; determining whether the target data warehouse needs to perform data archiving based on the archiving startup parameters includes: if the time interval between the current system date and the previous archiving date reaches the preset archiving period, and the current number of tasks in the target data warehouse is zero, then it is determined that the target data warehouse needs to perform data archiving.

[0079] Specifically, if the time interval between the current system date and the previous archive date reaches a preset archive period, the current task count of the target data warehouse is further obtained. If the current task count of the target data warehouse is zero, it is determined that the target data warehouse needs to perform data archiving, and the subsequent data archiving process is directly initiated. If the current task count of the target data warehouse is not zero, it can be re-checked every preset time interval to determine whether the current task count of the target data warehouse is zero, until the current task count of the target data warehouse is zero, at which point it is determined that the target data warehouse needs to perform data archiving. The preset time interval is not unique and can be freely configured on the terminal page according to actual needs. For example, in this embodiment, the preset time interval is set to 3 minutes.

[0080] In this embodiment, by performing task data checks on the target data warehouse before initiating data archiving, data operation chaos in the target data warehouse can be avoided. By setting a preset time interval to determine the current number of tasks in the target data warehouse, the smooth progress of data archiving can also be ensured.

[0081] S400: If the target data warehouse needs to be archived based on the archive startup parameters, determine the list of data to be archived in the target data warehouse based on the archive execution parameters.

[0082] The list of tables to be archived refers to the tables in the target data warehouse that require archiving operations, and can be determined based on the archiving execution parameters. Correspondingly, the archiving execution parameters may include the name or number of the list of tables to be archived, which can be configured by maintenance personnel on the terminal page.

[0083] Specifically, if the target data warehouse requires data archiving based on the archiving startup parameters, then the system searches through all the zipper tables in the target data warehouse for tables matching the name or number of the zipper table to be archived, and identifies these as the zipper tables to be archived. If no zipper table matching the name or number is found in the target data warehouse, it indicates that this target data warehouse does not require data archiving, and the process can be skipped to the next target data warehouse or terminated directly.

[0084] S600: Perform data opening and closing chaining processing on the zipper data in the archived zipper table that conforms to the preset opening and closing chaining rules to obtain the target zipper table.

[0085] It's understandable that each zipper entry in a zipper table should be in an open or closed state, and correspondingly, it should have an open and close date. However, during data transmission and processing in the target data warehouse, network issues or technical problems may cause data errors, such as zipper data lacking a close date. Furthermore, when archiving data in the target data warehouse, only zipper data prior to the current time point needs to be archived. Data currently in an open state needs to be truncated before archiving. This avoids directly archiving unclosed zipper data in the zipper table to be archived, which could lead to the loss of certain slices of data at certain points in time. Correspondingly, preset open / close rules are used to identify unclosed zipper data in the zipper table to be archived, and then perform open / close processing on this unclosed zipper data.

[0086] Specifically, the preset opening and closing rules can include conditions where the opening date of the zipper data is less than the current system date and there is no closing date, or conditions where the opening date of the zipper data is less than the current system date and the closing date is greater than the current system date. The subsequent opening and closing processing can either perform data re-closing processing on zipper data whose opening date is less than the current system date and there is no closing date, or perform data truncation processing on zipper data whose opening date is less than the current system date and the closing date is greater than the current system date. It can be understood that the resulting target zipper table is a list of zippers that need to be archived and for which there are no unclosed zipper data before the current time point.

[0087] Furthermore, before performing data opening and closing chaining, it can be determined whether there is any chained data in the chained list to be archived that matches the preset opening and closing chaining rules. For example, if it is determined that there is no chained data in a certain chained list to be archived that matches the preset opening and closing chaining rules, or if there is no chained data in any chained list to be archived that matches the preset opening and closing chaining rules, then the data opening and closing chaining process for that chained list to be archived can be skipped, or the data opening and closing chaining process for all chained lists to be archived can be skipped directly, thereby improving data archiving efficiency.

[0088] S800: Performs data archiving and cleanup processing on the target zipper table based on the archive execution parameters.

[0089] Specifically, data archiving and cleaning includes two parts: data archiving and data cleaning. Data archiving involves backing up and storing the zipper data that needs to be archived in the target zipper table based on the archiving execution parameters. Data cleaning involves clearing the backed-up zipper data to avoid the problem of a rapid decline in the data processing performance of the target data warehouse as the data volume accumulates.

[0090] Furthermore, there is no single way to back up and store zip data based on archiving execution parameters. For example, archiving execution parameters could include a split archiving method, where zip data can be fragmented and partitioned in parallel and exported to text files, which are then backed up and stored. Archiving execution parameters could also include multiple backup storage methods (such as near-line storage or offline storage), where zip data can be stored in different storage devices depending on the backup storage method. Archiving execution parameters could also include a near-line storage period, i.e., how much data prior to the start of the data archiving process requires near-line storage for backup.

[0091] The aforementioned data archiving method obtains the data archiving target parameters corresponding to the target data warehouse. These parameters include archiving start parameters and archiving execution parameters. If the target data warehouse requires data archiving based on the archiving start parameters, the method determines the list of zippered data to be archived within the target data warehouse based on the archiving execution parameters. Then, data opening and closing chaining is performed on the zippered data in the list that conforms to preset opening and closing chaining rules to obtain the target zippered data list. This avoids directly archiving the list of zippered data to be archived, which may contain unclosed chained data, thus preventing the loss of sliced ​​data at certain points in time. Finally, data archiving and cleanup are performed on the target zippered data list based on the archiving execution parameters. This automated data archiving and opening / closing chaining actions significantly reduce the workload and error rate of operations and maintenance personnel through a simple operation process, achieving automatic and effective data archiving.

[0092] In one embodiment, such as Figure 3 As shown, in S600, data opening and closing chaining processing is performed on the zipper data in the zipper table to be archived that conforms to the preset opening and closing chaining rules, including the following S620 to S640, wherein:

[0093] S620: Perform data re-closing processing on zipper data in the archived zipper table where the opening date is earlier than the current system date and there is no closing date.

[0094] The current system date is the system date corresponding to the start of data archiving processing. An open chain date less than the current system date indicates that the data in the linked list was already in an open chain state when data archiving processing started, while the absence of a closing chain date indicates that the linked list data encountered an error.

[0095] Correspondingly, in the above situation, it is necessary to perform data re-closing processing on this zipper data to avoid the loss of slice data at subsequent time points. In one embodiment, such as Figure 4 As shown, S620 includes the following S622 to S624, wherein:

[0096] S622: Set the closing date of zipper data whose opening date is less than the current system date and which does not have a closing date to the current system date. Specifically, the zipper data can be closed first, and its closing date can be padded to the time when the data archiving process was started, i.e., set to the current system date.

[0097] S624: For zipper data with an opening date less than the current system date and no closing date, add a new zipper data with an opening date of the current system date and a closing date of a preset maximum date. Specifically, after closing the zipper data, since this zipper data was closed during archiving, it needs to be reopened to avoid losing slice data at subsequent points in time. Therefore, a new zipper data is added based on the field information of the original zipper data, only its opening date is set to the current system date, and the closing date is filled in with the preset maximum date, while the other field information remains unchanged, thus achieving the data re-closing processing of the zipper data.

[0098] S640: Perform data truncation processing on zipper data in the archived zipper table where the opening date is less than the current system date and the closing date is greater than the current system date.

[0099] In this case, if the opening date is less than the current system date and the closing date is greater than the current system date, it indicates that the data in the linked list is already in an open state and not yet closed when the data archiving process is initiated. Correspondingly, in the above situation, the linked list data needs to be truncated to ensure that only linked list data before the current system date is archived during data archiving, thus avoiding the loss of slice data from subsequent time points.

[0100] Specifically, S640 includes: setting the closing date of zipper data whose opening date is less than the current system date and whose closing date is greater than the current system date to the current system date; and adding new zipper data whose opening date is the current system date based on the zipper data whose opening date is less than the current system date and whose closing date is greater than the current system date.

[0101] Understandably, we can first close the zipper data where the opening date is earlier than the current system date and the closing date is later than the current system date, modifying the closing date to the time when the data archiving process was started, i.e., setting it to the current system date. Then, after closing the zipper data, since this zipper data was closed during the archiving process, it needs to be reopened to avoid losing slice data from subsequent time points. Therefore, we can add a new zipper data based on the original zipper data's field information, setting only its opening date to the current system date, while keeping the other field information unchanged, thus achieving data truncation processing for the zipper data.

[0102] In one embodiment, such as Figure 5 As shown, S800 includes the following S820 to S840, wherein:

[0103] S820: Perform data archiving processing on zipper data in the target zipper table whose opening date is less than the current system date, based on the archiving execution parameters.

[0104] It is understood that the data archiving mentioned in this application refers to archiving data prior to the system date corresponding to the start of the data archiving process. Therefore, the execution object corresponding to the data archiving process step is the zipper data in the target zipper table whose opening date is less than the current system date.

[0105] Specifically, this step requires backing up and storing zipper data in the target zipper table whose opening date is earlier than the current system date. It is understood that the method for backup and storage is not unique and can be determined based on the archiving execution parameters configured by the maintenance personnel. For example, in one embodiment, the archiving execution parameters include the split archiving method and the near-line storage period. The split archiving method represents the method of partitioning and sharding the execution object, such as specifying partitioning and exporting according to preset fields in the target zipper table, or sharding and exporting the execution object according to a preset data volume, etc. The near-line storage period represents how long the data is before the start of the data archiving process, and the near-line storage method must be used for backup and storage.

[0106] Correspondingly, in one embodiment, S820 includes: according to the split archiving method, storing zipper data in the target zipper table whose time interval from the opening date to the current system date is less than or equal to the near-line storage period in near-line storage, and storing zipper data in the target zipper table whose time interval from the opening date to the current system date is greater than the near-line storage period offline.

[0107] It is understandable that the backup storage method in the archive execution parameters can include nearline storage and offline storage. In nearline storage, the exported text files will be stored as files on network storage or a dedicated data storage server for nearline retrieval. In offline storage, the exported text files will be stored as compressed files on network storage, and then subsequently burned to disc for offline storage.

[0108] Specifically, during data archiving, zipper data in the target zipper table whose time interval from the opening date to the current system date is less than or equal to the near-line storage period will be sharded and partitioned for near-line storage using a split archiving method. zipper data in the target zipper table whose time interval from the opening date to the current system date is greater than the near-line storage period will be sharded and partitioned for offline storage using a split archiving method.

[0109] S840: After completing the data archiving process, perform data cleanup on the zipper data in the target zipper table whose opening date is earlier than the current system date.

[0110] Specifically, the determination of whether data archiving has been completed can be achieved either by receiving a task completion callback from the server or by comparing the stored data with the data in the target linked list. After determining that data archiving has been completed, data cleanup can be performed on linked data in the target linked list whose opening date is earlier than the current system date. This prevents a rapid decline in the data processing performance of the target data warehouse due to the accumulation of data volume.

[0111] Furthermore, the method for performing data cleanup is not unique; it can be accomplished using a data clearing command corresponding to the target data warehouse, or it can be accomplished using a data overwrite command corresponding to the target data warehouse. In this embodiment, data cleanup is achieved by performing an overwrite operation on the zipper data in the target zipper table whose opening date is earlier than the current system date.

[0112] In one embodiment, such as Figure 6 As shown, after S200, the method also includes: S300: generating the archiving script corresponding to the data archiving target parameters. The archiving script includes the main program script, the data warehouse execution script, and the data archiving script.

[0113] In this context, the archiving script can be understood as the executable program corresponding to the data archiving mentioned in this application. Following the steps in the data archiving method, the archiving script may include a main program script, a data warehouse execution script, and a data archiving script. Specifically, the main program script is the executable program related to determining whether to initiate data archiving and calling other scripts; the data warehouse execution script is the executable program related to operations on the linked tables in the target data warehouse; and the data archiving script is the executable program related to exporting linked data from the target data warehouse. In essence, after obtaining the data archiving target parameters, an archiving script corresponding to the data archiving target parameters can be automatically generated, thereby implementing the data archiving method of this application.

[0114] Correspondingly, such as Figure 6 As shown, S400 may include: if it is determined from the archiving startup parameters that the target data warehouse needs to be archived, determining the zipper table to be archived in the target data warehouse through the main program script and based on the archiving execution parameters. S600 may include: performing data opening and closing chaining processing on the zipper data in the zipper table to be archived that conforms to preset opening and closing chaining rules through the data warehouse execution script to obtain the target zipper table. S800 may include: performing data archiving and cleaning processing on the target zipper table through the data archiving script, the data warehouse execution script, and based on the archiving execution parameters.

[0115] It is understood that the above scripts can be written in different programming languages, and those skilled in the art can choose any suitable method. For example, in this embodiment, the main program script can be written in Java. Since the target data warehouse is a data warehouse built on the Hadoop distributed system architecture, this type of data warehouse generally uses the Hive data warehouse tool for data extraction, transformation, and loading. Therefore, the data warehouse execution script is written in Hive SQL. In addition, the script for exporting the chained data from the target data warehouse can be implemented using Shell Script. Since Shell scripts are similar to batch processing in Windows / DOS, both pre-load various commands into a file for easy one-time execution, thus facilitating maintenance personnel in setting up, and are more powerful than batch processing in Windows and more efficient than programs edited with other programming programs.

[0116] In one embodiment, the data archiving method of this application further includes: obtaining the execution log file corresponding to the archived script; and analyzing and inspecting the execution log file.

[0117] Specifically, the execution log files corresponding to the archived scripts are retrieved and analyzed. This can be done after each step of an archived script is executed, or after all archived scripts have been executed, the execution log files for all archived scripts are retrieved and analyzed. During the analysis, the execution log files are checked for error messages, and the number of error messages and their corresponding erroneous execution steps are counted. Corresponding exception messages can then be generated to ensure timely handling of error messages.

[0118] In this embodiment, by monitoring the archived scripts in real time and recording the execution status of each script, not only can the correct execution order of the scripts be guaranteed and duplicate execution be avoided, but also timely and targeted processing can be carried out when abnormal information is triggered, so as to ensure that the data archiving can be completed smoothly.

[0119] In one embodiment, a detailed example is provided to illustrate the data archiving method provided in this application. The data archiving method provided in this embodiment mainly uses an executable program written in HQL, Shell, and Java to perform opening and closing chaining, archiving, and cleaning of chained data in the target data warehouse via a scheduled task. Specifically, it includes the following steps:

[0120] Step 1: The scheduled task initiation process first retrieves the task list from the Hadoop task queue corresponding to the target data warehouse, determining the number of tasks currently being executed and those awaiting execution. If the number of tasks in the task list is greater than zero, it indicates that there are tasks being executed. To ensure the smooth progress of the data warehouse cleanup process, this determination is performed every three minutes until the number of currently executing and pending tasks equals zero, at which point the process can proceed to Step 2.

[0121] Step 2: Based on the set cleanup date parameter (set via preset archiving date or preset archiving period, which can be set to midnight of the end of each year) and the opening / closing rules, query the number of data entries in each zipper table in the target data warehouse that need to be re-opened / closed. If the result is zero, it means that the current zipper table does not need to be re-opened / closed, skip to step 4. Otherwise, opening / closing is required, proceed to step 3.

[0122] Step 3: Based on the set pending cleanup date parameter and opening / closing rules, perform re-opening and closing operations on the current zipper table using an HQL script. If the opening date of a zipper is less than the pending cleanup date parameter and the closing date is greater than the pending cleanup date parameter, truncate the zipper data, change its closing date to the pending cleanup date parameter, and then regenerate a new zipper data with the opening date set to the pending cleanup date parameter and the closing date remaining unchanged. If the opening date of a zipper data is less than the pending cleanup date parameter and there is no closing date, perform a re-closing operation on the zipper data, set its closing date to the pending cleanup date parameter, and then regenerate a new zipper data with the opening date set to the pending cleanup date parameter and the closing date set to infinity. If the opening date of a zipper data is less than the pending cleanup date parameter and the closing date is less than or equal to the pending cleanup date parameter, or vice versa, no action is taken.

[0123] Step 4: The program calls a Shell script to export data from the target data warehouse's linked list table whose opening date is less than the current date, in shards and partitions, based on the date parameter to be cleaned and the open / closed chain rules. This data is then stored as text files on file storage. The linked data will be archived according to two processing levels: near-line and offline. Archived data for near-line storage will be stored as files on network storage or a dedicated data storage server for near-line queries. Archived data for offline storage will be stored as compressed files on network storage for later offline storage via disc burning.

[0124] Step 5: The program calls an HQL script to perform an overwrite operation on the chained table of the target data warehouse, retaining only data in the chained table whose opening date is greater than or equal to the current date parameter.

[0125] Step 6: The entire process and results of each task execution were monitored. A log file was generated for each task execution step, recording detailed information about the task execution. The logs can be previewed online.

[0126] Step 7: Seventh, after all tasks have been completed, the program will analyze and check the log file again to check for error messages. If no errors are found, the program will indicate successful processing; otherwise, it will fail and roll back to the incorrect step to correct the error.

[0127] This embodiment effectively addresses the performance degradation in data processing and querying caused by the accumulation of large amounts of data after prolonged operation in a distributed data warehouse. Without affecting the continuity of business data, it automates data archiving and open / close chain operations through parameter configuration. This simplified process significantly reduces the workload and error rate of maintenance personnel. It also allows non-specialized database maintenance personnel to perform database data archiving and maintenance according to business needs. Through annual close-chain and archiving operations, database data can be migrated to near-line and offline storage on an annual basis, thereby greatly reducing the resource consumption of the distributed database and lowering the hardware costs of the database.

[0128] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0129] Based on the same inventive concept, this application also provides a data archiving apparatus for implementing the data archiving method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data archiving apparatus embodiments provided below can be found in the limitations of the data archiving method described above, and will not be repeated here.

[0130] In one embodiment, such as Figure 7 As shown, a data archiving device is provided, including: a parameter acquisition module 710, a zipper list determination module 720, an open / closed chain processing module 730, and an archiving cleanup module 740, wherein:

[0131] The parameter acquisition module 710 is used to acquire the data archiving target parameters corresponding to the target data warehouse; the data archiving target parameters include archiving start parameters and archiving execution parameters.

[0132] The zipper table determination module 720 is used to determine the zipper table to be archived in the target data warehouse based on the archive execution parameters when it is determined from the archive startup parameters that the target data warehouse needs to be archived.

[0133] The open / closed chain processing module 730 is used to perform data open / closed chain processing on the zipper data in the zipper table to be archived that conforms to the preset open / closed chain rules, so as to obtain the target zipper table.

[0134] The archive cleanup module 740 is used to perform data archiving and cleanup processing on the target zipper table based on the archive execution parameters.

[0135] In this embodiment, data archiving target parameters corresponding to the target data warehouse are obtained. These parameters include archiving initiation parameters and archiving execution parameters. If the target data warehouse requires data archiving based on the archiving initiation parameters, the archived zipper table in the target data warehouse is determined based on the archiving execution parameters. Then, data opening and closing chaining is performed on the zipper data in the archived zipper table that conforms to preset opening and closing chaining rules to obtain the target zipper table. This avoids directly archiving the archived zipper table, which may contain unclosed zipper data, thus preventing the loss of slice data at certain points in time. Finally, data archiving and cleanup are performed on the target zipper table based on the archiving execution parameters. This automated data archiving and opening / closing actions significantly reduce the workload and error rate of maintenance personnel through a simple operation process, achieving automatic and effective data archiving.

[0136] In one embodiment, the archiving start parameters include a preset archiving period and the current number of tasks in the target data warehouse; the apparatus further includes:

[0137] The archiving judgment module is used to determine that the target data warehouse needs to be archived when the time interval between the current system date and the previous archiving date reaches the preset archiving period and the current number of tasks in the target data warehouse is zero.

[0138] In one embodiment, the open / closed chain processing module 730 includes:

[0139] The re-closing processing module is used to perform data re-closing processing on zipper data in the zipper table to be archived where the opening date is earlier than the current system date and there is no closing date.

[0140] The truncation module is used to truncate data in the zipper table to be archived, where the opening date is less than the current system date and the closing date is greater than the current system date.

[0141] In one embodiment, the re-closing chain processing module is further configured to set the closing date of zipper data whose opening date is less than the current system date and which does not have a closing date to the current system date; and to add new zipper data whose opening date is the current system date and whose closing date is a preset maximum date based on the zipper data whose opening date is less than the current system date and which does not have a closing date.

[0142] In one embodiment, the archive cleanup module 740 includes:

[0143] The data archiving module is used to perform data archiving processing on zipper data in the target zipper table whose opening date is less than the current system date, based on the archiving execution parameters.

[0144] The data cleaning module is used to perform data cleaning on zipper data in the target zipper table whose opening date is earlier than the current system date after the data archiving process is completed.

[0145] In one embodiment, the archive execution parameters include the split archive method and the nearline storage cycle;

[0146] The data archiving module is also used to perform near-line storage for zipper data in the target zipper table whose time interval from the opening date to the current system date is less than or equal to the near-line storage period, and to perform offline storage for zipper data in the target zipper table whose time interval from the opening date to the current system date is greater than the near-line storage period, according to the split archiving method.

[0147] In one embodiment, the apparatus further includes:

[0148] The script generation module is used to generate archiving scripts corresponding to the data archiving target parameters. The archiving scripts include the main program script, the data warehouse execution script, and the data archiving script.

[0149] The zipper table determination module 720 is also used to determine the zipper table to be archived in the target data warehouse by means of the main program script and based on the archive execution parameters when it is determined that the target data warehouse needs to be archived according to the archive startup parameters.

[0150] The open / closed chain processing module 730 is also used to execute data open / closed chain processing on the zipper data in the zipper table to be archived that conforms to the preset open / closed chain rules through the data warehouse execution script, so as to obtain the target zipper table;

[0151] The archive cleanup module 740 is also used to perform data archiving and cleanup processing on the target zipper table through data archiving scripts, data warehouse execution scripts, and based on archive execution parameters.

[0152] In one embodiment, the apparatus further includes:

[0153] The log acquisition module is used to obtain the execution log file corresponding to the archived script;

[0154] The log inspection module is used to analyze and inspect the execution log files.

[0155] Each module in the aforementioned data archiving device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data during the target data warehouse archiving process. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data archiving method.

[0157] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0160] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data archiving method characterized by, The method comprises: obtaining data archiving target parameters corresponding to a target data warehouse; the data archiving target parameters comprise archiving start parameters and archiving execution parameters; in a case where it is determined according to the archiving start parameters that the target data warehouse needs to perform data archiving, determining a to-be-archived zoned table in the target data warehouse based on the archiving execution parameters; performing data open-close chain processing on zoned data in the to-be-archived zoned table that meets preset open-close chain rules to obtain a target zoned table; performing data archiving and cleaning processing on the target zoned table based on the archiving execution parameters; wherein the data open-close chain processing on the zoned data in the to-be-archived zoned table that meets the preset open-close chain rules comprises: performing data reclose chain processing on zoned data in the to-be-archived zoned table that has an open chain date less than a current system date and does not have a close chain date; and performing data truncation processing on zoned data in the to-be-archived zoned table that has an open chain date less than the current system date and has a close chain date greater than the current system date.

2. The method of claim 1, wherein, the archiving start parameters comprise a preset archiving period and a current task quantity of the target data warehouse; determining according to the archiving start parameters that the target data warehouse needs to perform data archiving comprises: if a time interval between the current system date and a last archiving date reaches the preset archiving period and the current task quantity of the target data warehouse is zero, it is determined that the target data warehouse needs to perform data archiving.

3. The method of claim 1, wherein, the data reclose chain processing on the zoned data in the to-be-archived zoned table that has an open chain date less than the current system date and does not have a close chain date comprises: setting a close chain date of the zoned data that has an open chain date less than the current system date and does not have a close chain date as the current system date; and based on the zoned data that has an open chain date less than the current system date and does not have a close chain date, adding zoned data that has an open chain date of the current system date and a close chain date of a preset maximum date.

4. The method of claim 1, wherein, the data archiving and cleaning processing on the target zoned table based on the archiving execution parameters comprises: performing data archiving processing on zoned data in the target zoned table that has an open chain date less than the current system date based on the archiving execution parameters; after the data archiving processing is completed, performing data cleaning processing on the zoned data in the target zoned table that has an open chain date less than the current system date.

5. The method of claim 4, wherein, the archiving execution parameters comprise a split archiving mode and an online storage period; the data archiving processing on the zoned data in the target zoned table that has an open chain date less than the current system date based on the archiving execution parameters comprises: according to the split archiving mode, performing online storage on zoned data in the target zoned table that has a time interval from an open chain date to the current system date less than or equal to the online storage period, and performing offline storage on zoned data in the target zoned table that has a time interval from an open chain date to the current system date greater than the online storage period.

6. The method according to any one of claims 1 to 5, characterized in that, after the target data warehouse corresponding data archiving target parameters are obtained, the method further comprises: generating an archiving script corresponding to the data archiving target parameter, the archiving script including a main program script, a data warehouse execution script, and a data archiving script; the determining, based on the archiving execution parameter, of the to-be-archived zoned table in the target data warehouse comprises: determining, based on the archiving execution parameter, the to-be-archived zoned table in the target data warehouse through the main program script; the performing, on the zoned data in the to-be-archived zoned table that meets the preset open-close chain rule, of data open-close chain processing to obtain a target zoned table comprises: performing, through the data warehouse execution script, data open-close chain processing on the zoned data in the to-be-archived zoned table that meets the preset open-close chain rule to obtain a target zoned table; the performing, based on the archiving execution parameter, of data archiving and cleaning processing on the target zoned table comprises: performing, through the data archiving script and the data warehouse execution script, data archiving and cleaning processing on the target zoned table based on the archiving execution parameter.

7. The method of claim 6, wherein, The method further comprises: obtaining an execution log file corresponding to the archiving script; analyzing and checking the execution log file.

8. A data archiving apparatus, characterized by, The device comprises: a parameter obtaining module configured to obtain data archiving target parameters corresponding to a target data warehouse, the data archiving target parameters including archiving start parameters and archiving execution parameters; a zoned table determining module configured to determine, based on the archiving execution parameter, a to-be-archived zoned table in the target data warehouse in the case that the target data warehouse needs to perform data archiving according to the archiving start parameters; an open-close chain processing module configured to perform data open-close chain processing on zoned data in the to-be-archived zoned table that meets a preset open-close chain rule to obtain a target zoned table; an archiving and cleaning module configured to perform data archiving and cleaning processing on the target zoned table based on the archiving execution parameter; The open-close chain processing module comprises: a re-close chain processing module configured to perform data re-close chain processing on zoned data in the to-be-archived zoned table that has an open chain date smaller than a current system date and has no close chain date; a truncation processing module configured to perform data truncation processing on zoned data in the to-be-archived zoned table that has an open chain date smaller than the current system date and has a close chain date greater than the current system date.

9. The apparatus of claim 8, wherein, The archiving start parameters include a preset archiving period and a current task quantity of the target data warehouse; The device further comprises: an archiving determining module configured to determine that the target data warehouse needs to perform data archiving when a time interval between a current system date and a last archiving date reaches the preset archiving period and the current task quantity of the target data warehouse is zero.

10. The apparatus of claim 8, wherein, The reclosing chain processing module is further configured to set the closing date of the zipper data without the closing date and with the opening date less than the current system date as the current system date; and add zipper data with the opening date as the current system date and the closing date as a preset maximum date based on the zipper data without the closing date and with the opening date less than the current system date.

11. The apparatus of claim 8, wherein, The archiving and cleaning module comprises: a data archiving module configured to perform data archiving processing on the zipper data with the opening date less than the current system date in the target zipper table based on the archiving execution parameter; a data cleaning module configured to perform data cleaning processing on the zipper data with the opening date less than the current system date in the target zipper table after the data archiving processing is completed.

12. The apparatus of claim 11, wherein, The archiving execution parameter comprises a split archiving mode and an online storage period. The data archiving module is further configured to perform online storage on the zipper data with the time interval from the opening date to the current system date less than or equal to the online storage period and offline storage on the zipper data with the time interval from the opening date to the current system date greater than the online storage period in the target zipper table according to the split archiving mode.

13. The apparatus of any one of claims 8 to 12, wherein, The apparatus further comprises: a script generation module configured to generate an archiving script corresponding to the data archiving target parameter, the archiving script comprising a main program script, a data warehouse execution script and a data archiving script; The zipper table determination module is further configured to determine the to-be-archived zipper table in the target data warehouse based on the archiving execution parameter through the main program script in a case where it is determined that the target data warehouse needs to perform data archiving according to the archiving start parameter; The opening and closing chain processing module is further configured to perform data opening and closing chain processing on the zipper data meeting a preset opening and closing chain rule in the to-be-archived zipper table through the data warehouse execution script to obtain a target zipper table; The archiving and cleaning module is further configured to perform data archiving and cleaning processing on the target zipper table based on the archiving execution parameter through the data archiving script and the data warehouse execution script.

14. The apparatus of claim 13, wherein, The apparatus further comprises: a log acquisition module configured to acquire an execution log file corresponding to the archiving script; a log checking module configured to analyze and check the execution log file.

15. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

17. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images