KVM data storage processing method and system and continuous data point recovery method thereof

Abstract

The invention provides a KVM data storage processing method and system and a continuous data point recovery method thereof, and relates to the technical field of disaster recovery backup. In the storage processing method, a server program firstly initializes and creates a configuration information table, an initialization volume, a log volume group and a metadata structure, and after source data of a production end is received, corresponding volumes, tables and metadata are updated according to data types. According to the continuous data point recovery method, after a recovery volume is created, according to the relation between a recovery time point M0 and a latest time point H0, recovery at the latest or any time point is executed; and if the data logics are inconsistent, executing step-by-step rollback. According to the method, by combining the initialization volume and the log volume group and metadata management, efficient tracking and updating of the data are achieved, and data consistency and integrity are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of disaster recovery and backup, and relates to a KVM data storage processing method, system and continuous data point recovery method thereof. Background Art

[0002] In the field of modern information technology, KVM (Kernel-based Virtual Machine), as an open source virtualization technology, has become an important choice for building a virtualized environment. It allows multiple operating systems to run on the same physical server by simulating hardware, thereby improving resource utilization and flexibility. In this context, the optimization of data storage structure has become an important direction for improving virtual machine performance and reliability.

[0003] At present, like normal hosts, KVM virtual machines in cloud computing also face the risk of data loss and damage. For this situation, most users use the backup method, which is to fully back up the data of the KVM virtual machine at a certain time and create a snapshot. However, the backup method has a long backup time interval and a large amount of backup data storage. With the long-term operation of the disaster recovery system, the data storage volume will continue to increase, especially in a continuous data protection environment, the data growth rate will be very fast, and it is often impossible to track data efficiently, and the data consistency and integrity cannot be guaranteed.

[0004] Therefore, how to design a data storage structure that can track and manage data more efficiently while ensuring the consistency and integrity of the data is a technical problem that urgently needs to be solved. Summary of the invention

[0005] In order to solve the technical problems in the above-mentioned background technology, the present invention provides a KVM data storage and processing method, system and continuous data point recovery method thereof.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: In a first aspect, a KVM data storage processing method is provided, the method being applied to a server program and comprising the steps of: Configuration information table creation step, initialization creation of configuration information table; Data volume creation steps: Initialize and create initialization volume and log volume group; The metadata structure creation step is to initialize and create the metadata structure. The metadata structure includes log volume information, mirror table, log metadata file, write operation log and data block change log. The mirror table stores the latest state of the virtual machine and locates the log metadata file record of each data block; the log metadata file records the versions of all data blocks of the virtual machine at each time point; the data block change log uses bitmap to store the storage state of all data blocks of the virtual machine at time T; The data storage step receives and determines the source data type of the production end. If the source data is basic configuration and synchronization data, the configuration information table is updated first, and then the initialization volume and metadata structure are updated; if the source data is a data change record, the log volume group and metadata structure are directly updated.

[0007] In a second aspect, a KVM data storage and processing system is provided, the system being applied to a server program, comprising: A configuration information table creation module is used to initialize and create a configuration information table; The data volume creation module is used to initialize and create the initialization volume and log volume group; The metadata structure creation module is used to initialize and create the metadata structure. The metadata structure includes log volume information, mirror table, log metadata file, write operation log and data block change log. The mirror table stores the latest state of the virtual machine and locates the log metadata file record of each data block. The log metadata file records the versions of all data blocks of the virtual machine at each time point. The data block change log uses bitmap to store the storage state of all data blocks of the virtual machine at time T. The judgment update module is used to receive and judge the source data type of the production end. If the source data is basic configuration and synchronization data, the configuration information table is updated first, and then the initialization volume and metadata structure are updated; if the source data is a data change record, the log volume group and metadata structure are directly updated.

[0008] In a third aspect, a method for recovering continuous data points is provided, which utilizes the above-mentioned KVM data storage and processing method, and further includes the steps of: A recovery volume creation step is to receive a recovery instruction and create a recovery volume; Determine the execution step, determine whether the recovery time point M0 is greater than or equal to the latest time point H0, if so, execute the latest time point recovery step, if not, execute the arbitrary time point recovery step; The latest point-in-time recovery step traverses the mirror table to determine whether the latest content of the data block is in the log volume group or the initialization volume, and then reads the data from the log volume group or the initial volume and writes it to the recovery volume, wherein the log volume group is read through the log metadata file; The recovery step at any time point is to obtain the bitmap at time M0 in the data block change log, and then traverse the bitmap to determine whether the latest content of the data block is in the log volume group or the initialization volume, and then read the data from the log volume group or the initial volume and write it to the recovery volume. The reading of the log volume group is done by combining the mirror table and the log metadata file positioning, and the reading of the initialization volume is done by positioning the mirror table; In the step-by-step rollback procedure, when the internal data of the virtual machine after restoration at the latest time point or any time point is not in a logically consistent state, the data block position before the recovery time point M0 is located by writing the operation log and the mirror table, and then the log metadata file is traced back to determine the last valid change content of each data block, and then the corresponding data is read from the initialization volume or log volume according to the last valid change content, and overwritten and written into the recovery volume.

[0009] The beneficial effects of the present invention are: The present invention combines the initialization volume and log volume group to reasonably organize basic configuration and synchronization data and change data, and adopts the metadata management method of the metadata structure, which can efficiently track and update data; at the same time, the dynamic update of the configuration information table, metadata, initialization volume and log volume group ensures data consistency and integrity, and also lays the foundation for quickly locating and recovering required data during the data recovery process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 This is a schematic diagram of the KVM data storage and processing method in Example 1 of the present invention.

[0012] Figure 2 This is a schematic diagram of the data storage structure in Example 1 of the present invention.

[0013] Figure 3 This is a schematic diagram of the log metadata file structure in Example 1 of the present invention.

[0014] Figure 4 This is a flow chart of the KVM data storage and processing method in Embodiment 2 of the present invention.

[0015] Figure 5 This is a schematic diagram of the structure of the KVM data storage and processing system in Example 3 of the present invention.

[0016] Figure 6 This is a schematic diagram of the structure of the update module in Example 3 of the present invention.

[0017] Figure 7 Schematic diagram of the continuous data point recovery method in Example 4 of the present invention.

[0018] In the accompanying drawings, the components represented by the reference numerals are listed as follows: 3001, configuration information table creation module; 3002, data volume creation module; 3003, metadata structure creation module; 3004, judgment update module; 3005, initial volume write module; 3006, update bitmap data module; 3007, log volume deletion and log volume information module; 3008, metadata modification module; 30041, receiving unit; 30042, judgment unit; 30043, basic configuration and synchronization data storage unit; 30044, change data storage unit. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] Example 1

[0021] At present, like normal hosts, KVM virtual machines in cloud computing also face the risk of data loss and damage. For this situation, most users use the backup method, which is to fully back up and create snapshots of the data of the KVM virtual machine at a certain time. However, the backup method has a long backup time interval and a large amount of backup data storage. With the long-term operation of the disaster recovery system, the data storage volume will continue to increase, especially in the continuous data protection environment, the data growth rate will be very fast, and it is often impossible to track data efficiently, and the data consistency and integrity cannot be guaranteed. Therefore, how to design a data storage structure that can track and manage data more efficiently while ensuring data consistency and integrity is a technical problem that needs to be solved urgently.

[0022] In view of the above problems, an embodiment of the present invention provides a KVM data storage and processing method. Figure 1 A schematic diagram of a KVM data storage processing method provided by an embodiment of the present invention, combined with Figure 1 and Figure 2 As shown, the method includes: Step S101, initializing and creating a configuration information table.

[0023] It can be understood that the configuration information table stores the basic information of all virtual machines, including the unique identification code, total disk size and bitmap data path of the virtual machine. The configuration information table is arranged according to the unique identification code, and the record can be searched according to the unique identification code; the initialization volume and log volume group can be generated according to the total disk size; the bitmap data path stores the absolute path of the sent bitmap data on the local end, which can locate the bitmap data.

[0024] It can also be understood that the configuration information table locates the information position of the target virtual machine, which refers to whether the information of the target virtual machine is located in the initialization volume, the log volume group, or the metadata structure.

[0025] Step S102, initializing and creating an initialization volume and a log volume group.

[0026] It is understandable that since the management mechanism of directly operating block devices is relatively complex, this embodiment uses files to store data, improves the flexibility of data storage, and can operate block devices in a simpler way. The initialization volume stores the basic configuration and synchronization data of the virtual machine, and the size of the volume can be consistent with the total size of the virtual machine disk. The log volume group stores the change data generated by the production end in sequence, and the size of each log volume can be half of the total size of the virtual machine disk.

[0027] Step S103, initialize and create a metadata structure, the metadata structure includes log volume information, mirror table, log metadata file, write operation log and data block change log, wherein the mirror table stores the latest status of the virtual machine and locates the log metadata file record of each data block, and the log metadata file records the versions of all data blocks of the virtual machine at each time point; the data block change log uses a bitmap to store the storage status of all data blocks of the virtual machine at time T.

[0028] It is understandable that the write operation log stores every write operation information of the virtual machine. The write operation log increases dynamically as the virtual machine continues to write, and is stored in the order of write operations. Each record contains a timestamp, the starting logical block number of the write, and the number of data blocks written.

[0029] It is also understandable that the mechanism of data block change log is similar to the snapshot principle. After rolling back at any time point, the data volume type where the data block is stored can be directly queried based on the bitmap, minimizing the traversal of the log metafile. The data block change log record includes the timestamp of the bitmap creation and the absolute path of the bitmap.

[0030] The data block change log dynamically adds records according to the time interval set by the user, and writes the data block flag of the mirror table at that moment into the created bitmap, so that the records can be arranged and searched in the order of timestamps. Each value of the bitmap should be consistent with the data block flag value.

[0031] Optionally, in step S103, if Figure 3As shown, the log metadata file is a double linked list structure, each log metadata file record includes a head pointer head and a tail pointer last, the head pointer head points to the data block version that was changed for the first time and finds the tail pointer last, the tail pointer last points to the data block version that was changed most recently; the double linked list structure has a head node, a last node and a middle node; The mirror table includes a data block flag flag and a log metadata offset log_meta, wherein the log metadata offset log_meta is the offset of the log metadata of the data block recorded in the log metadata file, and the log metadata offset log_meta stores the address of the head pointer head.

[0032] It can be understood that the double linked list structure has a head node, a last node and a middle node. The double linked list supports bidirectional traversal, which can be from the head to the end of the table or from the end of the table to the head of the table.

[0033] It can also be understood that the log metadata file dynamically increases records as the virtual machine continues to write, and each record stores the versions of a data block at each time point. Unlike traditional log metadata files, each log metadata file record uses a bidirectional linked list structure to organize and store the versions of the data block at each time point. Of course, by traversing the log metadata file, it is possible to quickly restore to any continuous data point, thereby providing a guarantee for subsequent accelerated continuous recovery.

[0034] It is also worth pointing out that Figure 3 As shown, each node of the log metadata file contains a data domain, a pre pointer, and a next pointer. The pre pointer and the next pointer point to the previous and next nodes of the node respectively, so the log metadata record can be traversed in positive and reverse order respectively through the pre pointer and the next pointer. The data domain is a structure Data_Info, which contains the microsecond write timestamp of the data block, the log volume file name log_volume where the data block is written, and the offset within the log volume where the data block is written. Therefore, the position of the data block in the log volume can be located through the data domain of the node.

[0035] It is worth noting that the order of initializing and creating the configuration information table, the initialization volume, the log volume group, and the metadata structure can be improved in efficiency through parallel processing, but it is necessary to ensure that the dependency relationship is satisfied, that is, the configuration information table is dependent on the initialization volume and the log volume group, and the initialization volume and the log volume group are dependent on the metadata structure; they can also be created in sequence, such as initializing and creating the configuration information table, then creating the initialization volume and the log volume group, and finally creating the metadata structure. This embodiment does not make specific limitations on this.

[0036] Step S104, receiving and determining the source data type of the production end, if the source data is basic configuration and synchronization data, first update the configuration information table, then update the initialization volume and metadata structure; if the source data is a data change record, directly update the log volume group and metadata structure.

[0037] Optionally, step S102 includes: Step S1021, receiving source data from the production end; Step S1022, determining the type of source data, if the source data is configuration information, bitmap data and synchronization data without holes, then executing the basic configuration and synchronization data storage steps; if the source data is a data change record, then executing the change data storage step, wherein the data change record includes a unique identification code, a timestamp, an offset, a data length and a data content; Step S1023, for configuration information, first generate a configuration information table record according to the received configuration information and add it to the configuration information table, and then generate the data volume and metadata of the target virtual machine according to the generated record; for bitmap data, first initialize the metadata according to the bitmap data, and then write the absolute path of the bitmap data into the target virtual machine record in the configuration information table; for synchronization data without holes, first convert the synchronization data without holes and the bitmap data to obtain initialization synchronization data, and then write the initialization synchronization data into the data volume; Step S1024, search for the metadata and data volume of the virtual machine according to the unique identification code in the data change record, write the data content in the record into the log volume group, and then update the write operation log, update the mirror table, update the log metadata file and update the data block change log.

[0038] It is understandable that the configuration information stores virtual machine information, including the unique identification code of the virtual machine and the total size of the virtual disk file. The non-hole synchronization data is a non-hole version of the initialization synchronization data of the virtual machine, which can be converted into the initialization synchronization data by combining with the bitmap data. The bitmap data is the data block storage state of the initialization synchronization data of the virtual machine, which can be converted into the initialization synchronization data by combining with the non-hole synchronization data.

[0039] This embodiment combines the initialization volume and log volume group to reasonably organize the basic configuration and synchronization data and change data, and adopts the metadata management method of the metadata structure, which can efficiently track and update data; at the same time, the dynamic update of the configuration information table, metadata, initialization volume and log volume group ensures data consistency and integrity, and also lays the foundation for quickly locating and recovering required data during the data recovery process.

[0040] Example 2

[0041] like Figure 4As shown, a KVM data storage processing method is provided, and the method is applied to a server program, comprising the steps of: Step S201, initializing and creating a configuration information table; Step S202, initializing and creating an initialization volume and a log volume group, the log volume group includes multiple log volumes, and each log volume corresponds to log volume information in a metadata structure; Step S203, creating a metadata structure including log volume information, mirror table, log metadata file, write operation log and data block change log, wherein the mirror table stores the latest state of the virtual machine and locates the log metadata file record of each data block; The log metadata file records the versions of all data blocks of the virtual machine at each point in time, and also stores the storage location of all data in the log volume group; The data block change log uses bitmap to store the storage status of all data blocks of the virtual machine at time T; The log volume information includes the data block offset offset and the data block timestamp time, where the data block offset offset is the offset of each data block stored in the log volume in the initialization volume, and the data block timestamp time is the timestamp when the last data block stored in the log volume is written; Step S204, receiving and determining the source data type of the production end, if the source data is basic configuration and synchronization data, first update the configuration information table, then update the initialization volume and metadata structure; if the source data is a data change record, directly update the log volume group and metadata structure; Step S205, find the old log volume LV0 and its log volume information in the log volume group, traverse the data block offset offset of the log volume information in the log volume LV0, and then overwrite all data block contents in the log volume LV0 into the initialization volume according to the data block offset offset; Step S206, searching the configuration information table according to the unique identification code of the target virtual machine, obtaining the bitmap data of the target virtual machine, and then synchronously traversing the initialization volume and the bitmap data to update the bitmap data; Step S207, after all the data of log volume LV0 is written into the initialization volume, the log volume LV0 and its corresponding log volume information are deleted; Step S208, correspondingly modifying the mirror table, log metadata file, write operation log and data block change log.

[0042] Different from the above embodiment, this embodiment provides a storage space recovery processing strategy, which merges the virtual machine log volume data content into the initialization volume when the data storage capacity reaches the limit, and modifies and deletes the relevant metadata. This not only ensures the long-term stable operation of the system, but also improves the storage utilization efficiency, adapts to the demand for growing data volume, and enhances the scalability and sustainability of the system.

[0043] Example 3

[0044] like Figure 5 As shown, a KVM data storage and processing system is provided, and the system is applied to a server program, including: Configuration information table creation module 3001, used to initialize and create a configuration information table; The data volume creation module 3002 is used to initialize and create the initialization volume and log volume group.

[0045] The metadata structure creation module 3003 is used to initialize and create the metadata structure. The metadata structure includes log volume information, mirror table, log metadata file, write operation log and data block change log. The mirror table stores the latest status of the virtual machine and locates the log metadata file record of each data block. The log metadata file records the versions of all data blocks of the virtual machine at each time point. The data block change log uses a bitmap to store the storage status of all data blocks of the virtual machine at time T.

[0046] The judgment update module 3004 is used to receive and judge the source data type of the production end. If the source data is basic configuration and synchronization data, the configuration information table is updated first, and then the initialization volume and metadata structure are updated; if the source data is a data change record, the log volume group and metadata structure are directly updated.

[0047] Optionally, in the metadata structure creation module 3003, the log metadata file is a double linked list structure, and each log metadata file record includes a head pointer head and a tail pointer last, the head pointer head points to the data block version that was changed for the first time and finds the tail pointer last, and the tail pointer last points to the data block version that was changed most recently; The mirror table includes a data block flag flag and a log metadata offset log_meta, wherein the log metadata offset log_meta is the offset of the log metadata of the data block recorded in the log metadata file, and the log metadata offset log_meta stores the address of the head pointer head.

[0048] Optionally, in the data volume creation module 3002, the log volume group includes multiple log volumes, and each log volume corresponds to log volume information in a metadata structure; In the metadata structure creation module 3003, the log metadata file stores the storage location of all data in the log volume group; the log volume information includes a data block offset and a data block timestamp, wherein the data block offset is the offset of each data block stored in the log volume in the initialization volume, and the data block timestamp is the timestamp of the last data block stored in the log volume. Likewise Figure 5 As shown, the system further includes: The write initial volume module 3005 is used to find the old log volume LV0 and its log volume information in the log volume group, traverse the data block offset offset of the log volume information in the log volume LV0, and then overwrite all data block contents in the log volume LV0 into the initialization volume according to the data block offset offset; The bitmap data updating module 3006 is used to search the configuration information table according to the unique identification code of the target virtual machine, obtain the bitmap data of the target virtual machine, and then synchronously traverse the initialization volume and the bitmap data to update the bitmap data; The module 3007 for deleting the log volume and its log volume information is used to delete the log volume LV0 and its corresponding log volume information after writing all the data of the log volume LV0 into the initialization volume; Modify metadata module 3008, used to modify the mirror table, log metadata file, write operation log and data block change log like Figure 6 As shown, optionally, the determining update module 3004 includes: The receiving unit 30041 is used to receive source data from the production end; The judging unit 30042 is used to judge the type of the source data. If the source data is configuration information, bitmap data and synchronization data without holes, the basic configuration and synchronization data storage steps are performed; if the source data is a data change record, the change data storage step is performed, wherein the data change record includes a unique identification code, a timestamp, an offset, a data length and a data content; The basic configuration and synchronization data storage unit 30043 is used for, for configuration information, first generating a configuration information table record according to the received configuration information and adding it to the configuration information table, and then generating the data volume and metadata of the target virtual machine according to the generated record; for bitmap data, first initializing the metadata according to the bitmap data, and then writing the absolute path of the bitmap data into the target virtual machine record in the configuration information table; for synchronization data without holes, first converting the synchronization data without holes and the bitmap data into initialization synchronization data, and then writing the initialization synchronization data into the data volume; The change data storage unit 30044 is used to find the metadata and data volume of the virtual machine according to the unique identification code in the data change record, and then write the data content in the record into the log volume group, and then update the write operation log, update the mirror table, update the log metadata file and update the data block change log.

[0049] This embodiment integrates the initialization volume and the log volume group and implements data management based on the metadata structure, thereby building an efficient data tracking and update framework, optimizing the basic configuration and data synchronization process, and ensuring data consistency and integrity through a dynamic update mechanism.

[0050] Example 4

[0051] like Figure 7 As shown, a continuous data point recovery method is provided, using the KVM data storage processing method described in Example 1 or 2, including the steps of: Step S401, initializing and creating a configuration information table; Step S402, initializing and creating an initialization volume and a log volume group; Step S403, initializing and creating a metadata structure, which includes log volume information, mirror table, log metadata file, write operation log and data block change log, wherein the mirror table stores the latest state of the virtual machine and locates the log metadata file record of each data block, and the log metadata file records the versions of all data blocks of the virtual machine at each time point; the data block change log uses a bitmap to store the storage state of all data blocks of the virtual machine at time T; Step S404, receiving and determining the source data type of the production end, if the source data is basic configuration and synchronization data, first update the configuration information table, then update the initialization volume and metadata structure; if the source data is a data change record, directly update the log volume group and metadata structure; Step S405, receiving a recovery instruction and creating a recovery volume; Step S406, determine whether the recovery time point M0 is greater than or equal to the latest time point H0, if so, execute step S407, if not, execute step S108; Step S407, traversing the mirror table to determine whether the latest content of the data block is in the log volume group or the initialization volume, and then reading data from the log volume group or the initial volume and writing it into the recovery volume, wherein the log volume group is read through the log metadata file; Step S408, obtaining the bitmap at time M0 in the data block change log, and then traversing the bitmap to determine whether the latest content of the data block is in the log volume group or the initialization volume, and then reading data from the log volume group or the initial volume and writing it into the recovery volume, wherein reading the log volume group is performed by combining the mirror table and the log metadata file positioning, and reading the initialization volume is performed by positioning the mirror table; Step S409, when the internal data of the virtual machine after recovery at the latest time point or any time point is not in a logically consistent state, the data block position before the recovery time point M0 is located by writing the operation log and the mirror table, and then the log metadata file is traced back to determine the last valid change content of each data block, and then the corresponding data is read from the initialization volume or the log volume according to the last valid change content, and overwritten and written into the recovery volume.

[0052] It is understandable that before recovery, the server needs to create a recovery volume, generally setting the recovery volume data to zero, and the volume size during recovery is the same as the initial volume.

[0053] The latest time point H0 refers to the time point when the system successfully recorded the data change for the last time. It is the snapshot time point of the latest data status in the system and represents the latest version of the current data.

[0054] The recovery time point M0 refers to the target time point to which the data is expected to be recovered. This time point can be any moment in the past. It may be that due to data loss, damage or error, the data is expected to be recovered to this specific historical state.

[0055] In this embodiment, if the recovery time point M0 is greater than or equal to the latest time point H0, it means that the user wants to recover to a state containing the latest data, so the latest time point recovery step is performed. If the recovery time point M0 is less than the latest time point H0, the user wants to recover to an earlier state, so the arbitrary time point recovery step is performed.

[0056] It can be understood that the data block flag of the mirror table has three values: 0 means the latest content of the data block is in the log volume group, 1 means the latest content of the data block is in the initialization volume, and 2 means the data block is invalid data, that is, the block is an unallocated block or the data is zero. The log metadata offset log_meta is the offset of the log metadata of the data block in the log metadata file. The log metadata offset log_meta stores the address of the head pointer head In this embodiment, for the latest time point recovery, this embodiment traverses the mirror table, and checks the data block flag field of each record to determine whether the latest content of the data block is in the log volume group or the initialization volume. If the data block flag flag is 2, it can be skipped directly. If the data block flag flag is 1, the mirror table data block record is taken out, because the offset of the record in the mirror table is actually the offset of the data block in the initialization volume, the data can be directly read from the initialization volume according to the offset of the record in the mirror table. If the data block flag flag is 0, it is necessary to locate the log metadata file according to the log metadata offset log_meta, and the log metadata file records the versions of all data blocks of the virtual machine at each time point, and then read the log volume through the log metadata file. More specifically, the head node in the log metadata file is located according to the log metadata offset log_meta, and the last node is found through the pre pointer of the head node, and the log volume file name (log_volume) and the offset in the volume (offset) in the Data_Info structure are obtained, and the data is read from the log volume according to the log volume file name and the offset in the volume.

[0057] In this embodiment, for recovery at any time point, the data block change log uses bitmap to store the storage status of all data blocks of the virtual machine at a certain moment. After the time of rollback at any time point, the data volume type where the data block is stored can be directly queried according to the bitmap, and the record closest to the user-specified time point M0 can be quickly located by binary search. After determining the record closest to M0, the status of each data block can be quickly checked through the bitmap. If the bit value is 2, it means that the data block is invalid and skipped; if the bit value is 1, it means that the latest data of the data block is in the initialization volume; if the bit value is 0, it means that the data may be in the log volume group. If it is in the initialization volume, locate the mirror table record according to the offset in the bitmap, and read the data from the initialization volume; if it may be in the log volume group, locate the mirror table record according to the offset in the bitmap, and then locate the head node in the log metadata file according to the log metadata offset log_meta, traverse the log metadata record in reverse order, find the node with a timestamp less than or equal to M0, obtain the log volume file name and offset, and read the data from the log volume. If no qualified node is found, it means that the data is in the initialization volume.

[0058] In addition, gradual rollback recovery is a method to ensure that data is restored to a specified historical state, ensuring data consistency and integrity by gradually rolling back to the most recent consistent state.

[0059] In this embodiment, the write operation log records all write operations, and the mirror table provides the current status and location information of the data block. By combining these two parts, the last change content of the data block can be determined; the log metadata file records the versions of all data blocks of the virtual machine at each time point, and the data status at a specified time point can be found by reverse traversal; and by gradually rolling back and overwriting the write recovery volume, it can be ensured that the restored data is consistent with the status at the specified time point, and even in the case of data inconsistency, it can be handled by rollback.

[0060] More specifically, use the binary search method to locate the recovery time point M0 whose timestamp is less than or equal to M0 and the most recent write operation record. According to the starting block number and number of blocks in the write operation log, search the mirror table record to determine the head node in the log metadata file of each data block. Traverse the log metadata records in reverse order to find the node with a timestamp less than or equal to M0, and determine the last change content of the data block.

[0061] If the previous node of the target node is null, the data is stored in the initialization volume; if it is not null, the data is stored in the log volume. If the data is stored in the initialization volume, read the data directly from the initialization volume according to the offset in the image table; if the data is stored in the log volume, read the data from the log volume according to the log volume file name and offset in the Data_Info structure of the previous node. After that, the data read from the initialization volume or log volume is overwritten and written to the recovery volume according to the offset in the image table. If the internal data of the restored virtual machine is inconsistent, continue to roll back and repeat the above operation.

[0062] This embodiment supports multiple recovery methods, including latest time point recovery, arbitrary time point recovery and gradual rollback recovery, which can meet different business needs. When data loss or error occurs, users can choose the most appropriate recovery plan according to the actual situation. By traversing the mirror table and log metadata files, the required data can be accurately restored from the initialization volume and log volume group, ensuring rapid recovery in different failure situations.

[0063] Computer storage media for storing embodiments of the present invention may be any combination of one or more computer-readable media. Computer-readable media may be computer-readable signal media or computer-readable storage media. Computer-readable storage media may be, for example, but not limited to, systems, devices or components of electricity, magnetism, light, electromagnetics, infrared rays, or semiconductors, or any combination thereof. More specific examples (non-exhaustive lists) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, computer-readable storage media may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device, or device or used in combination therewith.

[0064] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0065] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0066] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Ruby, Go, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0067] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A KVM data storage and processing method, characterized in that: The method is applied to a server program and comprises the steps of: Configuration information table creation step, initialization creation of configuration information table; Data volume creation steps: Initialize and create initialization volume and log volume group; The metadata structure creation step is to initialize and create the metadata structure. The metadata structure includes log volume information, mirror table, log metadata file, write operation log and data block change log. The mirror table stores the latest state of the virtual machine and locates the log metadata file record of each data block; the log metadata file records the versions of all data blocks of the virtual machine at each time point. The data block change log uses bitmap to store the storage status of all data blocks of the virtual machine at time T; Data storage step: receiving and determining the source data type of the production end. If the source data is basic configuration and synchronization data, the configuration information table is updated first, and then the initialization volume and metadata structure are updated; If the source data is a data change record, the log volume group and metadata structure are directly updated.

2. The KVM data storage and processing method according to claim 1, characterized in that: In the metadata structure creation step, the log metadata file is a double linked list structure, and each log metadata file record includes a head pointer head and a tail pointer last, the head pointer head points to the data block version that was changed for the first time and finds the tail pointer last, and the tail pointer last points to the data block version that was changed most recently; The mirror table includes a data block flag flag and a log metadata offset log_meta, wherein the log metadata offset log_meta is the offset of the log metadata of the data block recorded in the log metadata file, and the log metadata offset log_meta stores the address of the head pointer head.

3. The KVM data storage and processing method according to claim 1, characterized in that: In the data volume creation step, the log volume group includes a plurality of log volumes, and each log volume corresponds to log volume information in a metadata structure; In the metadata structure creation step, the log metadata file stores the storage location of all data in the log volume group; the log volume information includes a data block offset and a data block timestamp, wherein the data block offset is the offset of each data block stored in the log volume in the initialization volume, and the data block timestamp is the timestamp when the last data block stored in the log volume is written.

4. The KVM data storage and processing method according to claim 3, characterized in that: The method further comprises the steps of: In the step of writing the initial volume, find the old log volume LV0 and its log volume information in the log volume group, traverse the data block offset offset of the log volume information in the log volume LV0, and then overwrite all the data block contents in the log volume LV0 with the initialization volume according to the data block offset offset; The step of updating the bitmap data is to search the configuration information table according to the unique identification code of the target virtual machine, obtain the bitmap data of the target virtual machine, and then synchronously traverse the initialization volume and the bitmap data to update the bitmap data; The step of deleting the log volume and its log volume information is to write all the data of the log volume LV0 into the initialization volume and then delete the log volume LV0 and its corresponding log volume information; The metadata modification step corresponds to modifying the mirror table, log metadata file, writing operation log and data block change log.

5. The KVM data storage and processing method according to claim 1, characterized in that: The step of determining and updating comprises: Receiving step, receiving source data from the production end; A judgment step, judging the type of source data, if the source data is configuration information, bitmap data and synchronization data without holes, then executing the basic configuration and synchronization data storage steps; if the source data is a data change record, then executing the change data storage step, wherein the data change record includes a unique identification code, a timestamp, an offset, a data length and a data content; Basic configuration and synchronization data storage steps: for configuration information, first generate a configuration information table record according to the received configuration information and add it to the configuration information table, and then generate the data volume and metadata of the target virtual machine according to the generated record; for bitmap data, first initialize the metadata according to the bitmap data, and then write the absolute path of the bitmap data into the target virtual machine record in the configuration information table; for synchronization data without holes, first convert the synchronization data without holes and the bitmap data to obtain the initialization synchronization data, and then write the initialization synchronization data into the data volume; The step of storing changed data is to search for the metadata and data volume of the virtual machine according to the unique identification code in the data change record, write the data content in the record into the log volume group, and then update the write operation log, update the mirror table, update the log metadata file and update the data block change log.

6. A KVM data storage and processing system, characterized in that: The system is applied to a server program, including: A configuration information table creation module is used to initialize and create a configuration information table; The data volume creation module is used to initialize and create the initialization volume and log volume group; The metadata structure creation module is used to initialize and create the metadata structure. The metadata structure includes log volume information, mirror table, log metadata file, write operation log and data block change log. The mirror table stores the latest state of the virtual machine and locates the log metadata file record of each data block. The log metadata file records the versions of all data blocks of the virtual machine at each time point. The data block change log uses bitmap to store the storage state of all data blocks of the virtual machine at time T. The data storage module determines the update module, which is used to receive and determine the source data type of the production end. If the source data is basic configuration and synchronization data, the configuration information table is updated first, and then the initialization volume and metadata structure are updated; if the source data is a data change record, the log volume group and metadata structure are directly updated.

7. The KVM data storage and processing system according to claim 6, characterized in that: In the metadata structure creation module, the log metadata file is a double-linked list structure, and each log metadata file record includes a head pointer head and a tail pointer last, the head pointer head points to the data block version that was changed for the first time and finds the tail pointer last, and the tail pointer last points to the data block version that was changed most recently; The mirror table includes a data block flag flag and a log metadata offset log_meta, wherein the log metadata offset log_meta is the offset of the log metadata of the data block recorded in the log metadata file, and the log metadata offset log_meta stores the address of the head pointer head.

8. The KVM data storage and processing system according to claim 6, characterized in that: In the data volume creation module, the log volume group includes multiple log volumes, and each log volume corresponds to log volume information in a metadata structure; In the metadata structure creation module, the log metadata file stores the storage location of all data in the log volume group; the log volume information includes a data block offset and a data block timestamp, wherein the data block offset is the offset of each data block stored in the log volume in the initialization volume, and the data block timestamp is the timestamp of the last data block stored in the log volume. The system further comprises: The write initial volume module is used to find the old log volume LV0 and its log volume information in the log volume group, traverse the data block offset offset of the log volume information in the log volume LV0, and then overwrite all data block contents in the log volume LV0 into the initialization volume according to the data block offset offset; The bitmap data updating module is used to search the configuration information table according to the unique identification code of the target virtual machine, obtain the bitmap data of the target virtual machine, and then synchronously traverse the initialization volume and the bitmap data to update the bitmap data; The module for deleting the log volume and its log volume information is used to delete the log volume LV0 and its corresponding log volume information after writing all the data of the log volume LV0 into the initialization volume; The metadata modification module is used to modify the image table, log metadata file, write operation log and data block change log.

9. The KVM data storage and processing system according to claim 6, characterized in that: The determining and updating module comprises: A receiving unit, used for receiving source data from the production end; A judgment unit, used to judge the type of source data, if the source data is configuration information, bitmap data and synchronization data without holes, then execute the basic configuration and synchronization data storage steps; if the source data is a data change record, then execute the change data storage step, wherein the data change record includes a unique identification code, a timestamp, an offset, a data length and a data content; The basic configuration and synchronization data storage unit is used to generate a configuration information table record according to the received configuration information and add it to the configuration information table, and then generate the data volume and metadata of the target virtual machine according to the generated record; for bitmap data, first initialize the metadata according to the bitmap data, and then write the absolute path of the bitmap data into the target virtual machine record in the configuration information table; for synchronization data without holes, first convert the synchronization data without holes and the bitmap data to obtain the initialization synchronization data, and then write the initialization synchronization data into the data volume; The change data storage unit is used to find the metadata and data volume of the virtual machine according to the unique identification code in the data change record, write the data content in the record into the log volume group, and then update the write operation log, update the mirror table, update the log metadata file and update the data block change log.

10. A method for recovering continuous data points, characterized in that: The KVM data storage processing method according to any one of claims 1 to 5 further comprises the steps of: A recovery volume creation step is to receive a recovery instruction and create a recovery volume; Determine the execution step, determine whether the recovery time point M0 is greater than or equal to the latest time point H0, if so, execute the latest time point recovery step, if not, execute the arbitrary time point recovery step; The latest point-in-time recovery step traverses the mirror table to determine whether the latest content of the data block is in the log volume group or the initialization volume, and then reads the data from the log volume group or the initial volume and writes it to the recovery volume, wherein the log volume group is read through the log metadata file; The recovery step at any time point is to obtain the bitmap at time M0 in the data block change log, and then traverse the bitmap to determine whether the latest content of the data block is in the log volume group or the initialization volume, and then read the data from the log volume group or the initial volume and write it to the recovery volume. The reading of the log volume group is done by combining the mirror table and the log metadata file positioning, and the reading of the initialization volume is done by positioning the mirror table; In the step-by-step rollback procedure, when the internal data of the virtual machine after restoration at the latest time point or any time point is not in a logically consistent state, the data block position before the recovery time point M0 is located by writing the operation log and the mirror table, and then the log metadata file is traced back to determine the last valid change content of each data block, and then the corresponding data is read from the initialization volume or log volume according to the last valid change content, and overwritten and written into the recovery volume.

Images