Data archiving method, system, device and storage medium

Abstract

One or more embodiments of this specification provide a data archiving method, system, electronic device, and computer-readable storage medium. The method includes: in response to a data archiving request, determining a data table to be archived in a source database and creating a temporary table in the source database; executing a database transaction in the source database, wherein executing the database transaction includes backing up the data in the data table to be archived to the temporary table and deleting the data in the data table to be archived after the backup is complete; and archiving the data in the temporary table to a target storage system after the database transaction is committed. This embodiment ensures data security in data archiving scenarios.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of terminal technology, and in particular, to a data archiving method, system, electronic device, and computer-readable storage medium. Background Art

[0002] Database archiving technology is a method for maintaining a relatively constant online database size while providing stable database performance for user applications. Database archiving involves moving inactive data to a separate storage device for long-term preservation. Specifically, database archiving involves first backing up the data to be archived to a separate storage device and then deleting the backed-up data from the database.

[0003] However, due to the time difference between the data backup process and the data deletion process, the following problem may occur: after backing up the data to other storage devices, during the process of deleting the backed-up data in the database, the user may modify the backed-up data in the database, and the system may delete the data modified by the user, resulting in inconsistency between the deleted data and the backed-up data. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data archiving method, system, electronic device, and computer-readable storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data archiving method is provided, including:

[0007] In response to a data archiving request, determining a data table to be archived in a source database, and creating a temporary table in the source database;

[0008] Executing a database transaction in the source database, wherein the executing the database transaction includes backing up the data in the to-be-archived data table to the temporary table, and deleting the data in the to-be-archived data table after the backup is completed;

[0009] After the database transaction is committed, the data in the temporary table is archived to a target storage system.

[0010] Optionally, the method further includes:

[0011] Slice the data in the to-be-archived data table to obtain at least two data slices;

[0012] The executing of the database transaction in the source database includes:

[0013] A database transaction is executed for any data slice to back up the data slice to the temporary table, and the data slice in the to-be-archived data table is deleted after the backup is completed.

[0014] Optionally, archiving the data in the temporary table to a target storage system includes:

[0015] The data slices backed up in the temporary table are concurrently archived into the target storage system.

[0016] Optionally, archiving the data in the temporary table to a target storage system includes:

[0017] According to the pre-stored conversion relationship between the storage structure of the source database and the storage structure of the target storage system, the data in the temporary table is converted, and the converted data is written into the target storage system.

[0018] Optionally, after archiving the data in the temporary table to the target storage system, the method further includes:

[0019] An end-to-end data verification method is used to verify whether the data archived in the target storage system is consistent with the data in the temporary table.

[0020] Optionally, after archiving the data in the temporary table to the target storage medium system, the method further includes:

[0021] When the number of days for which the temporary table is kept does not exceed the preset number of days, in response to a recovery request for the data table to be archived, performing data recovery processing according to the temporary table;

[0022] When the number of days the temporary table is kept exceeds a preset number of days, the temporary table is deleted.

[0023] Optionally, the target storage system is selected from a plurality of candidate storage systems based on at least one of the following information: expected storage cost, expected archiving efficiency, and usage requirements of the data table to be archived; and / or

[0024] The target storage system includes other databases and / or file storage systems with lower storage costs than the source database.

[0025] Optionally, archiving the data in the temporary table to a target storage system includes:

[0026] According to different usage requirements of different parts of the data in the temporary table, the different parts of the data are archived into different target storage systems.

[0027] According to a second aspect of one or more embodiments of this specification, a data archiving system is provided, including a source database, a data archiving device, and a target storage system, wherein the data archiving device is connected to the source database and the target storage system respectively;

[0028] The data archiving device is used to execute the method described in any one of the first aspects.

[0029] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0030] processor;

[0031] a memory for storing processor-executable instructions;

[0032] The processor implements the method as described in any one of the first aspects by running the executable instructions.

[0033] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0034] The embodiments of this specification have the following beneficial effects:

[0035] An embodiment of the present specification provides a data archiving method. After receiving a data archiving request, a temporary table can be created in a source database where a data table to be archived is located. A database transaction is then executed in the source database. The executing the database transaction includes backing up the data in the data table to be archived to the temporary table, and deleting the data in the data table to be archived after the backup is completed, thereby ensuring that the data backed up in the temporary table is consistent with the deleted data based on the characteristics of the database's local transactions. After the database transaction is committed, the data in the temporary table is archived to a target storage system. The temporary table is data that is invisible to the user, and the user cannot modify the data in the temporary table. By creating a temporary table in the source database, executing a local transaction in the source database, and then archiving the data in the temporary table, the effect of "deleting data first and then backing up data" is simulated, thereby ensuring the consistency of deleted data and backup data during the data archiving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of an application scenario provided by an exemplary embodiment.

[0037] Figure 2 This is a flowchart of a data archiving method provided by an exemplary embodiment.

[0038] Figure 3 It is a schematic diagram of processing data shards provided by an exemplary embodiment.

[0039] Figure 4 It is a structural diagram of a device provided by an exemplary embodiment.

[0040] Figure 5 It is a block diagram of a data archiving device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0042] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0043] Database archiving is the process of moving data that is no longer frequently used to a separate storage device for long-term storage. Specifically, database archiving involves first backing up the data to be archived to a separate storage device and then deleting the backed-up data from the database.

[0044] However, due to the time difference between the data backup process and the data deletion process, the following problem may occur: after backing up the data to other storage devices, during the process of deleting the backed-up data in the database, the user may modify the backed-up data in the database, and the system may delete the data modified by the user, resulting in inconsistency between the deleted data and the backed-up data.

[0045] In response to the problems in the related art, an embodiment of the present specification provides a data archiving method. After receiving a data archiving request, a temporary table can be created in the source database where the data table to be archived is located; then a database transaction is executed in the source database, and the execution of the database transaction includes backing up the data in the data table to be archived to the temporary table, and deleting the data in the data table to be archived after the backup is completed, thereby ensuring that the data backed up in the temporary table is consistent with the deleted data based on the characteristics of the database local transaction; and then after the database transaction is committed, the data in the temporary table is archived to the target storage system; wherein the temporary table is data that is invisible to the user, and the user cannot modify the data in the temporary table; this embodiment realizes the method of creating a temporary table in the source database and executing a local transaction in the source database, and then archiving the data in the temporary table, simulating the effect of "deleting data first and then backing up data", thereby ensuring the consistency of deleted data and backup data during the data archiving process.

[0046] A database transaction is a series of operations performed as a single logical unit of work in a database. Local database transactions have four major characteristics:

[0047] (1) Atomicity: A transaction is executed as a whole. All operations on the database contained in it are either executed in full or not. Therefore, if the transaction operation succeeds, it must be fully applied to the database. If the operation fails, it will not have any impact on the database.

[0048] (2) Consistency: Consistency means that a transaction must cause the database to change from one consistent state to another. In other words, the database must be in a consistent state before and after a transaction is executed. For example, if the total amount of money between user A and user B is 5,000, then no matter how A and B transfer money or how many times, the total amount of money between the two users should still be 5,000 after the transaction is completed. This is the consistency of the transaction.

[0049] (3) Isolation: Isolation means that when multiple users access the database concurrently, for example, when operating on the same table, the transaction opened by the database for each user cannot be interfered with by the operations of other transactions. Multiple concurrent transactions must be isolated from each other. In other words, the following effect should be achieved: for any two concurrent transactions T1 and T2, from the perspective of transaction T1, T2 either ends before T1 starts or starts after T1 ends. In this way, each transaction is unaware of the concurrent execution of other transactions.

[0050] (4) Durability: Durability means that once a transaction is committed, the changes to the data in the database are permanent, and the transaction commit operation will not be lost even if the database system encounters a failure.

[0051] In some embodiments, the data archiving methods provided in the embodiments of this specification can be performed by electronic devices. Exemplarily, the electronic devices include, but are not limited to, physical servers, virtual servers, cloud servers, personal computers, laptop computers, cellular phones, camera phones, smartphones, personal digital assistants, media players, navigation devices, email transceiver devices, game consoles, tablet computers, wearable devices, or any combination of these devices.

[0052] Exemplarily, the electronic device includes a processor and a memory; the memory stores instructions executable by the processor; the processor executes the executable instructions to implement the data archiving method provided in this embodiment. Exemplarily, the data archiving method can be a computer program product integrated into the electronic device.

[0053] For example, see Figure 1 The electronic device can establish a connection with the source database and the target storage system, thereby backing up and deleting data in the source database, and writing data in the temporary table of the source database into the target storage system.

[0054] In some embodiments, see Figure 2 , Figure 2 This is a flow chart of a data archiving method provided in an embodiment of this specification. The method can be performed by an electronic device and includes:

[0055] In step S201 , in response to a data archiving request, a data table to be archived in a source database is determined, and a temporary table is created in the source database.

[0056] Illustratively, the data archiving request carries information about the data table to be archived in this data archiving process, so that the electronic device can determine the data table to be archived in the source database based on the information carried in the data archiving request.

[0057] In step S202 , a database transaction is executed in the source database. The executing of the database transaction includes backing up the data in the to-be-archived data table to the temporary table, and deleting the data in the to-be-archived data table after the backup is completed.

[0058] In step S203, after the database transaction is committed, the data in the temporary table is archived to the target storage system.

[0059] In this embodiment, after receiving a data archiving request, a temporary table can be created in the source database where the data table to be archived is located, and then a database transaction is executed in the source database, so as to ensure that the data backed up in the temporary table is consistent with the deleted data based on the characteristics of the local transaction of the database; and then after the database transaction is committed, the data in the temporary table is archived to the target storage system; wherein, the temporary table is data that is invisible to the user, and the user cannot modify the data in the temporary table; by creating a temporary table in the source database and executing a local transaction in the source database, and then archiving the data in the temporary table, the effect of "deleting data first and then backing up data" is simulated, so that the deleted data and the backup data are consistent during the data archiving process, and the data security in the database archiving scenario is effectively guaranteed.

[0060] The aforementioned data archiving request can be a timed request triggered based on a preset archiving cycle. For example, the device hosting the source database can periodically determine the usage frequency of each data table stored in the source database, then identify data tables with usage frequencies below a preset frequency threshold as data tables to be archived, and send a data archiving request to the electronic device. Alternatively, the data archiving request can be triggered by the source database management program or user, or by detecting a specific change in the data in the source database. This specification does not specifically limit this.

[0061] After determining the data table to be archived in the source database, the electronic device may create a temporary table in the source database; wherein the table structure of the temporary table is the same as the table structure of the data table to be archived, so as to improve the backup efficiency in the subsequent backup process; after creating the temporary table, the electronic device may execute a database transaction in the source database, wherein executing the database transaction includes backing up the data in the data table to be archived to the temporary table and deleting the data in the data table to be archived after the backup is completed; since the table structure of the temporary table is the same as the table structure of the data table to be archived, no additional format conversion processing is required for the data during the data backup process, thereby facilitating improved backup efficiency; and further, after the database transaction is committed, the data in the temporary table may be archived to the target storage system.

[0062] In some embodiments, in order to enable data archiving to be performed without the user's awareness, after creating a temporary table in the source database, the data in the data table to be archived can be sharded to obtain at least two data shards. Then, a database transaction is executed for any one of the at least two data shards to back up the data shard to the temporary table, and the data shard in the data table to be archived is deleted after the backup is completed. During the execution of the database transaction for any data shard, the user cannot modify the data shard, thereby ensuring that the data shard to be backed up is consistent with the deleted data shard based on the characteristics of the database local transaction. Then, the electronic device can concurrently archive the data shards backed up in the temporary table to the target storage system, thereby improving data archiving efficiency; for example, the electronic device can speed up the archiving speed of the data in the data table to be archived by reasonably allocating multi-threaded resources.

[0063] For an example, see Figure 3 For a total of 40,000 data items with IDs ranging from 1 to 40,000 in the data table to be archived, the electronic device can perform sharding processing on the data table to be archived, for example, Figure 4 The data table to be archived is divided into four data shards according to four preset sharding conditions (id≥1andid<10024, id≥10024andid<21024, id≥21024andid<29501, and id≥29501andid≤40000), and then a database transaction is executed for each data shard to back up the data shard to the temporary table. After the backup is completed, the data shard in the data table to be archived is deleted, and the consistency of the backed-up data shards and the deleted data shards is ensured by utilizing the characteristics of local transactions of the database; and then the data shards backed up to the temporary table are concurrently archived to the target storage system. Exemplarily, the target storage system may be a target database having a storage cost lower than that of a source database. The electronic device may establish a target table in the target database, and then archive the data in the temporary table of the source database to the target table in the target database.

[0064] In some embodiments, considering the differences between the storage structure of the source database and the storage structure of the target storage system, the electronic device can pre-store the conversion relationship between the storage structures of different databases and the storage structures of different storage systems; and then, when archiving data in a temporary table, the data in the temporary table can be converted according to the pre-stored conversion relationship between the storage structure of the source database and the storage structure of the target storage system, and the converted data can be written into the target storage system.

[0065] The data archiving method provided in this specification is universal, and the steps of the data archiving method provided in the embodiments of this specification can be used to archive data in all databases. For example, an electronic device can support archiving between multiple different source databases and multiple different target storage systems. For example, an electronic device can reuse code in a source database or target storage system through inheritance to quickly support new source databases or new target storage systems.

[0066] In some embodiments, it is considered that data corruption may occur at any stage in the data archiving process, such as (1) hardware errors during the archiving process, such as failures of memory, CPU (central processing unit), network card, etc.; (2) signal-noise interference during data transmission; (3) firmware problems, such as failures of RAID (Redundant Arrays of Independent Disks) controllers, disk controllers, etc.; (4) software problems, such as operating system kernel problems, local file system problems, network system problems, I / O (Input / Output) scheduling layer problems, etc.

[0067] Therefore, after archiving the temporary table in the source database, the electronic device can use an end-to-end data verification method to verify whether the data archived in the target storage system is consistent with the data in the temporary table, in preparation for the subsequent deletion of the temporary table. The end-to-end data verification method means that when the application layer writes data, it calculates and adds a checksum information at each link, and sends this checksum information and the data block to the disk. After receiving the data packet, the disk will recalculate a new checksum information and compare it with the received checksum information. If the comparison is inconsistent, it is considered that an error occurred in the archiving process, and the data archived in the target storage system is inconsistent with the data in the temporary table; conversely, if the comparison is consistent, it can be considered that the data archived in the target storage system is consistent with the data in the temporary table. For example, if the verification is consistent, the temporary table in the source database can be deleted; if the verification is inconsistent, the process of archiving the data in the temporary table to the target storage system can be re-executed to ensure data consistency and data security in the data archiving scenario.

[0068] In some embodiments, the target storage system may include other databases and / or file storage systems with lower storage costs than the source database. For example, in a cloud-native data warehouse scenario, the file storage system may be an OSS (Object Storage Service) storage system deployed in the cloud.

[0069] Exemplarily, the target storage system may be specified by the user, for example, the data archiving request may carry information about the target storage system. During the data archiving process, the electronic device may directly archive the data in the temporary table to the target storage system indicated by the data archiving request.

[0070] Exemplarily, the target storage system can be selected by the electronic device from multiple candidate storage systems; for example, the data archiving request can carry at least one of the following information: the user's expected storage cost, expected archiving efficiency, and usage requirements of the data table to be archived. The electronic device can select the target storage system from multiple candidate storage systems based on at least one of the expected storage cost, expected archiving efficiency, and usage requirements of the data table to be archived.

[0071] For example, the storage costs of the multiple candidate storage systems may differ, and the multiple candidate storage systems may include databases and file storage systems. For example, the storage cost of a database with both read / write and data analysis capabilities may be higher than the storage cost of a file storage system with only read capabilities. Furthermore, different databases may have different storage costs due to differences in read / write performance and / or data analysis performance. The electronic device may select a target storage system from the multiple candidate storage systems based on the user's expected storage cost and the difference between the reference storage costs of the to-be-archived data tables in the candidate storage systems.

[0072] For example, the multiple candidate storage systems are located in different regions. The communication distances between the candidate storage systems in different regions and the source database are different, and thus the archiving efficiency is also different. The communication distance is negatively correlated with the archiving efficiency: the longer the communication distance, the lower the archiving efficiency; conversely, the shorter the communication distance, the higher the archiving efficiency. The electronic device can select a target storage system from the multiple candidate storage systems based on the user's desired archiving efficiency and the communication distances between each candidate storage system and the source database. For example, if the user prioritizes archiving efficiency and hopes to complete data archiving as quickly as possible, the candidate storage system with the shorter communication distance to the source database can be selected as the target storage system from the multiple candidate storage systems.

[0073] For example, based on different actual application scenarios, the data tables to be archived may have different usage requirements; for example, some data tables to be archived only need to be viewed in the subsequent application process, while some data to be archived may need to be analyzed, or some data to be archived may need to be modified in the subsequent application process; and different storage systems can provide different functions, for example, a file storage system can only provide a data viewing function (i.e., a reading function), while a database can provide data reading and writing functions and data analysis functions. Therefore, the electronic device can select a target storage system from multiple candidate storage systems based on the usage requirements of the data tables to be archived and the functions provided by each candidate storage system. For example, if the data tables to be archived have a read and write requirement in the subsequent application process, the selected target storage system can be a database; for another example, if the data tables to be archived only have a viewing requirement in the subsequent application process, the selected target storage system can be a file storage system; for another example, if the data tables to be archived have a simple data analysis requirement in the subsequent application process, the selected target storage system can be a database with a lower storage cost and certain data analysis functions.

[0074] For example, considering that different parts of the data in the data table to be archived may have different usage requirements, the electronic device can determine the different target storage systems corresponding to different parts of the data in the temporary table according to the usage requirements of the data in the data table to be archived and the functions that can be provided by each candidate storage system in multiple candidate storage systems; and then, after backing up the data in the data table to be archived to the temporary table, the different parts of the data can be archived to different target storage systems according to the usage requirements of the data in different parts of the temporary table, thereby realizing personalized data archiving requirements.

[0075] For example, if part of the data in the data table to be archived requires data analysis in the subsequent application process, while the other part of the data only requires viewing in the subsequent application process, the part of the data with data analysis requirements can be archived to the database, and the part of the data with viewing requirements can be archived to the file storage system to meet the user's personalized archiving needs.

[0076] In some embodiments, after archiving the data in the to-be-archived data table to the target storage system, the electronic device also needs to archive the metadata of the to-be-archived data table to the target storage system. Metadata is data about data, primarily describing the data's properties, and is used to support functions such as indicating storage location, historical data, resource search, and file logging.

[0077] In some embodiments, after archiving the data in the temporary table to the target storage system, in order to meet the user's data recovery needs, the temporary table can be saved in the source database table for a preset number of days. It can be understood that the specific value of the preset number of days can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on this. In the case that the storage number of days of the temporary table does not exceed the preset storage number of days, data recovery processing can be performed according to the temporary table in response to the recovery request of the data table to be archived; for example, in the case that the table structure of the temporary table is the same as the table structure of the data table to be archived, the temporary table can be used as the data table to be archived by modifying the metadata of the temporary table; in the case that the storage number of days of the temporary table exceeds the preset storage number of days, the temporary table can be deleted, thereby saving storage resources of the source database.

[0078] In a possible implementation, after the data in the to-be-archived data table is archived to the target storage system, the user can directly access the to-be-archived data table in the target storage system through the client without going through the source database.

[0079] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0080] Figure 4 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410. Of course, it may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0081] Please refer to Figure 5 , the data archiving device can be used for Figure 4 The device shown in the figure is used to implement the technical solution of this specification. The data archiving device may include:

[0082] The request response module 501 is configured to determine a data table to be archived in a source database in response to a data archiving request, and to create a temporary table in the source database;

[0083] A database transaction execution module 502 is configured to execute a database transaction in the source database, wherein the executing of the database transaction includes backing up the data in the to-be-archived data table to the temporary table and deleting the data in the to-be-archived data table after the backup is completed.

[0084] The archiving module 503 is configured to archive the data in the temporary table to the target storage system after the database transaction is committed. In some embodiments, the system further includes a sharding module configured to shard the data in the to-be-archived data table to obtain at least two data shards. The database transaction execution module 502 is specifically configured to execute a database transaction on any data shard to back up the data shard to the temporary table and, after the backup is complete, delete the data shard from the to-be-archived data table.

[0085] In some embodiments, the archiving module 503 is specifically configured to concurrently archive the data slices backed up in the temporary table to the target storage system.

[0086] In some embodiments, the archiving module 503 is specifically used to convert the data in the temporary table according to the pre-stored conversion relationship between the storage structure of the source database and the storage structure of the target storage system, and write the converted data into the target storage system.

[0087] In some embodiments, a verification module is further included, which is used to verify whether the data archived in the target storage system is consistent with the data in the temporary table using an end-to-end data verification method.

[0088] In some embodiments, a recovery and deletion module is further included, which is used to respond to a recovery request of the data table to be archived and perform data recovery processing according to the temporary table when the number of days the temporary table is kept does not exceed the preset number of days; and to delete the temporary table when the number of days the temporary table is kept exceeds the preset number of days.

[0089] In some embodiments, the target storage system is selected from multiple candidate storage systems based on at least one of the following information: expected storage cost, expected archiving efficiency, and usage requirements of the data table to be archived; and / or the target storage system includes other databases and / or file storage systems with lower storage costs than the source database.

[0090] In some embodiments, the archiving module 503 is specifically configured to archive different parts of the data in the temporary table into different target storage systems according to different usage requirements of the data in different parts.

[0091] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0092] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0093] Correspondingly, the present disclosure also provides a data archiving system, including a source database, the above-mentioned data archiving device and a target storage system, wherein the data archiving device is connected to the source database and the target storage system respectively; the data archiving device is used to execute any one of the methods described above.

[0094] For example, the data archiving device may be installed in Figure 1 or Figure 4 In the electronic device shown.

[0095] Accordingly, the present disclosure also provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the steps in any one of the above methods.

[0096] Accordingly, the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0098] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0099] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0100] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0101] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0102] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0103] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and concurrent processing are also possible or may be advantageous.

[0104] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0105] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0106] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data archiving method, comprising: In response to a data archiving request, determining a data table to be archived in a source database, and creating a temporary table in the source database; Executing a database transaction in the source database, wherein the executing the database transaction includes backing up the data in the to-be-archived data table to the temporary table, and deleting the data in the to-be-archived data table after the backup is completed; After the database transaction is committed, the data in the temporary table is archived to the target storage system; wherein, at each stage of writing data, the application layer calculates and adds a checksum information, and sends the checksum information and the data block to the disk; After receiving the data packet, the disk recalculates the new checksum information and compares the new checksum information with the received checksum information; If the comparison is inconsistent, it is determined that the data archived in the target storage system is inconsistent with the data in the temporary table; and the process of archiving the data in the temporary table to the target storage system is re-executed; If the comparison is consistent, it is determined that the data archived in the target storage system is consistent with the data in the temporary table, and the temporary table in the source database is deleted; The method further comprises: Slice the data in the to-be-archived data table to obtain at least two data slices; The executing of the database transaction in the source database includes: Execute a database transaction for any data shard to back up the data shard to the temporary table, and delete the data shard in the to-be-archived data table after the backup is completed; Archiving the data in the temporary table to the target storage system includes: Convert the data in the temporary table according to the pre-stored conversion relationship between the storage structure of the source database and the storage structure of the target storage system, and write the converted data into the target storage system; After archiving the data in the temporary table to the target storage medium system, the method further includes: When the number of days for which the temporary table is kept does not exceed the preset number of days, in response to a recovery request for the data table to be archived, performing data recovery processing according to the temporary table; When the number of days the temporary table is kept exceeds a preset number of days, the temporary table is deleted.

2. The method according to claim 1, wherein archiving the data in the temporary table to the target storage system comprises: The data slices backed up in the temporary table are concurrently archived into the target storage system.

3. The method according to claim 1 or 2, further comprising, after archiving the data in the temporary table to the target storage system: An end-to-end data verification method is used to verify whether the data archived in the target storage system is consistent with the data in the temporary table.

4. The method according to claim 1 or 2, wherein the target storage system is selected from a plurality of candidate storage systems based on at least one of the following information: expected storage cost, expected archiving efficiency, and usage requirements of the data table to be archived; and / or The target storage system includes other databases and / or file storage systems with lower storage costs than the source database.

5. The method according to claim 1 or 2, wherein archiving the data in the temporary table to a target storage system comprises: According to different usage requirements of different parts of the data in the temporary table, the different parts of the data are archived into different target storage systems.

6. A data archiving system, comprising a source database, a data archiving device, and a target storage system, wherein the data archiving device is connected to the source database and the target storage system respectively; The data archiving device is used to execute the method according to any one of claims 1 to 5.

7. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 5 by running the executable instructions.

8. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images