Data update log generation method, data backup method, device, and medium

Abstract

The present disclosure provides a data update log generation method, a data backup method, an apparatus and a medium. The method comprises: in response to starting of a virtual machine, obtaining a backup state of data written by the virtual machine to a storage space of a host. In response to receiving at least one write request from the virtual machine, for each write request, determining a target subspace for storing payload data carried by the write request from a plurality of subspaces based on the write request, and storing the payload data in the target subspace. Determining that the backup state of the payload data in the target subspace is not backed up to a server, and generating a data update log of the target subspace based on the write request, the data update log being used to represent a data update situation of the target subspace. Packaging and sending the data update logs of the plurality of write requests to the server, so that the server backs up the data update logs. The method takes into account the data backup accuracy of the virtual machine and the normal operation of the virtual machine.

Description

Technical Field

[0001] This disclosure relates to the field of computer data protection technology, and in particular to a data update log generation method, data backup method, device and medium. Background Technology

[0002] As enterprises fully migrate their core businesses to the cloud, virtual machines, as an important carrier of these businesses, are crucial for data backup. Data backup affects the data reliability and business continuity of core businesses.

[0003] In related technologies, virtual machine data is backed up using timed snapshots. This involves triggering snapshot operations at set intervals, briefly pausing the virtual machine's operations at a preset time point, creating a copy of the disk data, and saving it as a snapshot file to record the complete data state at that moment.

[0004] However, the relevant technologies have the problem of not being able to balance the accuracy of virtual machine data backups with the normal operation of virtual machine services. Summary of the Invention

[0005] In view of this, this disclosure provides a method for generating data update logs, a method for data backup, an apparatus, and a medium.

[0006] One aspect of this disclosure provides a data update log generation method, comprising: in response to virtual machine startup, obtaining the backup status of data written by the virtual machine to the host machine's storage space, the storage space including multiple subspaces, the backup status including backed up to the server or not backed up to the server; in response to receiving at least one write request from the virtual machine, for each write request, determining a target subspace from the multiple subspaces for storing payload data carried by the write request, and storing the payload data in the target subspace; determining the backup status of the payload data in the target subspace as not backed up to the server, and generating a data update log for the target subspace based on the write request, the data update log being used to characterize the data update status of the target subspace; and packaging the data update logs of multiple write requests and sending them to the server so that the server can back up the data update logs.

[0007] According to embodiments of this disclosure, the backup status of data in each subspace is obtained when the virtual machine starts, providing a basis for subsequent determination of which data needs to be backed up. A write request is used to determine the target subspace where payload data needs to be stored, and the payload data is then stored, thus implementing the virtual machine's write request. The payload data in the target subspace is marked as unbacked up, and a data update log is generated to facilitate marking data update status. Multiple data update logs are packaged and sent to the server to achieve incremental backup and reduce transmission overhead.

[0008] According to embodiments of this disclosure, the data update log generation method further includes: obtaining the starting addresses of multiple subspaces when the backup status of the data in the subspace is not backed up to the server; sending the data of the multiple subspaces to the server sequentially based on the starting addresses; and updating the backup status of the data to be backed up to the server.

[0009] According to embodiments of this disclosure, when a subspace's backup status is "not backed up to the server," the starting address of each subspace is obtained to determine the data backup order. Based on the starting address, the data from each subspace is sequentially sent to the server, and the backup status is updated to "backed up to the server," thus achieving a full data backup.

[0010] According to embodiments of this disclosure, generating a data update log for a target subspace based on a write request includes: determining a sequence number representing the generation time of the data update log based on the write request; generating the data update log based on the sequence number and the data offset, data length, and payload data obtained by parsing the write request, wherein the data offset indicates the starting address for storing the payload data in the storage space.

[0011] According to embodiments of this disclosure, a sequence number is determined based on the write request to characterize the generation time of the data update log, facilitating the management of the log's timing. The data update log is generated based on the sequence number and the data offset, data length, and payload data obtained from parsing the write request, thereby accurately recording the write request.

[0012] According to embodiments of this disclosure, the data update log generation method further includes: obtaining the increment of the number of subspaces not backed up to the server obtained from each scan, wherein the scan includes scanning from the initial address to the end address of the storage space; when the number of subspaces not backed up to the server in the backup state is less than a preset threshold or the increments of the number of subspaces are the same for two consecutive scans, switching the state of the host machine from a synchronization state to a stable state, and sending a first notification to the server, indicating that the data written by the virtual machine to the storage space of the host machine has been sent to the server.

[0013] According to embodiments of this disclosure, the increment of the number of unbacked-up subspaces obtained from each scan is acquired to understand the changing trend of unbacked-up data. If the number of unbacked-up subspaces is less than a preset threshold or if two consecutive increments are the same, the host machine state is switched from synchronization state to stable state, indicating that full data synchronization is nearing completion. A first notification is then sent to the server, indicating that the data in the storage space has been sent to the server, so that the server can confirm that full synchronization is complete.

[0014] According to embodiments of this disclosure, the data update log generation method further includes: when the host machine switches to a stable state, if the cumulative duration from the last time multiple data update logs were packaged and sent to the current time is greater than or equal to a preset threshold, or the number of multiple data update logs is greater than or equal to a number threshold, the multiple data update logs are packaged and sent to the server.

[0015] According to embodiments of this disclosure, a stable host state indicates that the data in the storage space has been fully synchronized, and attention only needs to be paid to data updates caused by write requests. Therefore, logs need to be packaged and sent subsequently to synchronize the server. The timing for triggering packaged sending is accurately determined by whether the cumulative time since the last packaged sending of data update logs reaches a preset threshold or whether the number of logs reaches a certain threshold. When the conditions are met, multiple data update logs are packaged and sent to the server, achieving incremental backup in a stable state and avoiding the overhead of frequent transmissions.

[0016] According to embodiments of this disclosure, the data update log generation method further includes: storing multiple data update logs in a first preset buffer of the host machine; when the data volume of the multiple data update logs is greater than the capacity of the first preset buffer, storing the multiple data update logs overflowing the first preset buffer in a second preset buffer of the host machine, wherein the data update logs in the second preset buffer are sent to the server with a higher priority than the data update logs in the first preset buffer; and generating alarm information when the data volume of the multiple data update logs is greater than a capacity threshold, wherein the capacity threshold is determined based on the capacity of the first preset buffer and the capacity of the second preset buffer.

[0017] According to embodiments of this disclosure, data update logs are stored in a first preset buffer on the host machine for temporary log management. When the log data volume exceeds the capacity of the first preset buffer, the overflowing logs are stored in a second preset buffer, and the log transmission priority of the second preset buffer is higher than that of the first preset buffer, thereby ensuring priority transmission of overflowing data. When the log data volume exceeds a capacity threshold determined by the capacities of the two buffers, an alarm message is generated to promptly alert the risk of insufficient storage resources.

[0018] According to embodiments of this disclosure, the data update log generation method further includes: in response to receiving a disk flushing instruction or a forced unit access instruction from a virtual machine, sending a second notification to a server, indicating that the payload data of the write request sent from the virtual machine startup to the current time has been stored in the physical hard disk.

[0019] According to embodiments of this disclosure, in response to a disk flushing command or a forced unit access command from a virtual machine, a second notification is sent to the server, indicating that the write request payload data from the virtual machine startup to the current time has been stored on the physical hard disk, so that the server knows that the time point corresponding to the second notification can be used as a recoverable time point and thus recover the data before that time point.

[0020] One aspect of this disclosure provides a data backup method applied to a server, characterized in that the method includes: in response to receiving a data update log from a host machine, backing up the data update log, wherein the data update log is generated using a data update log generation method.

[0021] Another aspect of this disclosure provides a data update log generation apparatus, comprising: a backup status acquisition module, configured to acquire, in response to virtual machine startup, the backup status of data written by the virtual machine to the host machine's storage space, the storage space including multiple subspaces, the backup status including backed up to the server or not backed up to the server; a target subspace determination module, configured to, in response to receiving at least one write request from the virtual machine, determine, for each write request, a target subspace for storing payload data carried by the write request from the multiple subspaces, and store the payload data in the target subspace; a data update log generation module, configured to determine the backup status of the payload data in the target subspace as not backed up to the server, and generate a data update log for the target subspace based on the write request, the data update log being used to characterize the data update status of the target subspace; and a log sending module, configured to package and send the data update logs of multiple write requests to the server, so that the server can back up the data update logs.

[0022] Another aspect of this disclosure provides a data backup apparatus, including: a log backup module, configured to back up the data update log in response to receiving a data update log from a host machine, wherein the data update log is generated using a data update log generation method.

[0023] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0024] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0025] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0027] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0028] Figure 1 The diagram illustrates the architecture of a data update log generation method according to an embodiment of the present disclosure.

[0029] Figure 2 A flowchart illustrating a data update log generation method according to an embodiment of the present disclosure is shown schematically.

[0030] Figure 3 A flowchart illustrating a data update log generation method according to another embodiment of this disclosure is shown schematically;

[0031] Figure 4 A flowchart illustrating a data update log generation method according to another embodiment of this disclosure is shown schematically;

[0032] Figure 5 A flowchart illustrating a host state transition process according to an embodiment of the present disclosure is shown schematically.

[0033] Figure 6 This schematic diagram illustrates a structural block diagram of an apparatus according to an embodiment of the present disclosure;

[0034] Figure 7 A schematic block diagram of an electronic device that can be used to implement the methods of embodiments of the present disclosure is shown. Detailed Implementation

[0035] The embodiments of this disclosure will now be described with reference to the accompanying drawings. Various details of the embodiments of this disclosure are included to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0037] In related technologies, virtual machine data backup is mainly carried out in three ways: First, by installing an agent and loading a kernel driver in the guest operating system of the virtual machine to intercept input / output (I / O), but this method has the problems of high maintenance costs and impact on virtual machine performance; Second, by backing up virtual machine data through periodic snapshots, the virtual machine's operations are briefly paused at preset time points to create a copy of the disk data and save it as a snapshot file, but if the snapshot interval is too long, data changes during the interval cannot be captured, thus increasing the risk of data loss, and the snapshot creation process is usually accompanied by the virtual machine's I / O pause, such instantaneous business interruption will affect the stable operation of sensitive services; Third, asynchronous data replication is performed at the storage layer, but there is the problem of losing newly written data.

[0038] This shows that the relevant technologies cannot simultaneously ensure the accuracy of virtual machine data backup and the normal operation of virtual machine services.

[0039] Figure 1 The diagram illustrates the architecture of a data update log generation method according to an embodiment of the present disclosure.

[0040] like Figure 1 As shown, the architecture includes virtual machine 110, host machine 120 and server 130.

[0041] In some embodiments, the architecture also includes a cloud platform management plane, through which users manage filters via the application programming interface (API) of the filters.

[0042] Virtual machine 110 is used for business processing, specifically utilizing the Guest OS on a kernel-based Virtual Machine (KVM) for business processing. Virtual machine 110 sends write requests to host machine 120 to write data into the host machine 120's storage space. In practical applications, there can be multiple virtual machines 110.

[0043] Host machine 120 includes a block storage chain, which comprises a VirtIO block device (virtio-blk), a throttle, a filter, a virtual disk, and a physical hard disk. The VirtIO block device serves as the entry point between virtual machine 110 and host machine 120; all read and write requests from virtual machine 110 enter host machine 120 through this entry point. The throttle is used to limit the rate of full synchronization or I / O. The filter is used to execute the data update log generation method disclosed herein. The virtual disk is the storage space of host machine 120, storing data written by virtual machine 110. The physical hard disk is the actual physical device of host machine 120, used to store data from the virtual disk.

[0044] The host machine 120 also includes a first preset buffer, which can be an unlocked loop buffer used to store data update logs.

[0045] The host machine 120 sends data and data update logs to the server 130 through the network layer.

[0046] Server 130 includes a log storage engine, a point-in-time recovery index module, and an image synthesis and recovery module. The log storage engine receives data update logs from the host machine and stores them in sequence according to their serial numbers. The point-in-time recovery index module generates an index record for each second notification time point, recording its corresponding sequence number range, timestamp, and log file offset. Multiple index records corresponding to the second notification time points form a timeline, from which the user can select any time point for recovery. The image synthesis and recovery module is used to assemble the data obtained from full synchronization into a complete basic disk image. It also retrieves the basic image and sequentially replays the data update logs before that time point according to the user's specified time point, synthesizing complete data to achieve data recovery.

[0047] Figure 2 A flowchart illustrating a data update log generation method according to an embodiment of the present disclosure is shown schematically.

[0048] like Figure 2 As shown, the flowchart 200 of this embodiment includes operations S210 to S240.

[0049] In operation S210, in response to the virtual machine startup, the backup status of the data written by the virtual machine to the host machine's storage space is obtained.

[0050] The storage space includes multiple subspaces.

[0051] It should be understood that storage space can be a virtual disk. The data written by the virtual machine to the host machine is written to the virtual disk of the host machine. The virtual disk is a file on the host machine's physical hard disk used to store the virtual machine's data.

[0052] Backup status includes whether it has been backed up to the server or not.

[0053] In practical applications, a bitmap data structure corresponding to the virtual disk capacity is created in the host machine's memory. This virtual disk is the virtual machine's storage file on the host machine. Each bit in the bitmap corresponds to a fixed-size data block on the virtual disk, with granularity configurable, set to 64 kilobytes (KB) or 1 megabyte (MB). Initially, all bits in the bitmap are marked as "dirty," indicating that they have not been backed up to the server. Each subspace can correspond to each data block of the virtual disk.

[0054] In operation S220, in response to receiving at least one write request from the virtual machine, for each write request, a target subspace for storing the payload data carried by the write request is determined from multiple subspaces based on the write request, and the payload data is stored in the target subspace.

[0055] It should be understood that the write request carries the target address of the storage payload data, i.e., the target subspace.

[0056] In operation S230, the backup status of the payload data in the target subspace is determined to be not backed up to the server, and a data update log for the target subspace is generated based on the write request.

[0057] It should be understood that when a virtual machine's services write I / O to the host machine, new payload data is generated. Therefore, the backup status of this payload data needs to be set to "not backed up to the server" to facilitate subsequent backups. In practical applications, the bitmap of the target subspace is marked as "dirty" to ensure that when the area is scanned subsequently, the latest data that has been overwritten by the service is reread, rather than outdated old data.

[0058] The data update log is used to characterize the data update status of the target subspace.

[0059] In operation S240, the data update logs of multiple write requests are packaged and sent to the server so that the server can back up the data update logs.

[0060] According to embodiments of this disclosure, the backup status of data in each subspace is obtained when the virtual machine starts, providing a basis for subsequent determination of which data needs to be backed up. A write request is used to determine the target subspace where payload data needs to be stored, and the payload data is then stored, thus implementing the virtual machine's write request. The payload data in the target subspace is marked as unbacked up, and a data update log is generated to facilitate marking data update status. Multiple data update logs are packaged and sent to the server to achieve incremental backup and reduce transmission overhead.

[0061] In some exemplary embodiments, the data update log generation method further includes: obtaining the starting addresses of multiple subspaces when the backup status of the data in a subspace is not backed up to the server; sequentially sending the data of the multiple subspaces to the server based on the starting addresses; and updating the backup status of the data to be backed up to the server.

[0062] In practical applications, Figure 1 In this embodiment, the filter initiates a coroutine for full synchronization. Based on the starting address, it sequentially reads all data from the virtual disk in ascending order of address. After reading data from each subspace, it sends the data to the server and simultaneously updates the backup status of that data to "backed up to the server," marking the bitmap corresponding to that subspace as "synchronized." To avoid excessive consumption of storage I / O bandwidth during full synchronization, this coroutine supports configurable I / O bandwidth limiting and I / O priority degradation.

[0063] According to embodiments of this disclosure, when a subspace's backup status is "not backed up to the server," the starting address of each subspace is obtained to determine the data backup order. Based on the starting address, the data from each subspace is sequentially sent to the server, and the backup status is updated to "backed up to the server," thus achieving a full data backup.

[0064] In some exemplary embodiments, generating a data update log for a target subspace based on a write request includes: determining a sequence number to characterize the time of data update log generation based on the write request; and generating the data update log based on the sequence number and the data offset, data length, and payload data obtained by parsing the write request.

[0065] It should be understood that in order for the server to store data update logs in the chronological order of write requests, sequence numbers are generated for differentiation, thus satisfying the first-come, first-served principle. The filter internally maintains a counter, which increments by 1 for each intercepted write request. Its current value is the sequence number of that log entry, which is globally unique and monotonically increasing within a single virtual disk dimension.

[0066] The data offset indicates the starting address for storing the payload data in the storage space. In practical applications, the data offset is the starting sector offset of the write operation on the virtual disk. The data length is the size of the data in the write operation, which can be in bytes.

[0067] The data payload represents the actual data content of the write operation.

[0068] According to embodiments of this disclosure, a sequence number is determined based on the write request to characterize the generation time of the data update log, facilitating the management of the log's timing. The data update log is generated based on the sequence number and the data offset, data length, and payload data obtained from parsing the write request, thereby accurately recording the write request.

[0069] In some exemplary embodiments, the data update log generation method further includes: obtaining the increment of the number of subspaces occupied by data not backed up to the server obtained from each scan. If the number of subspaces not backed up to the server is less than a preset threshold or the increments are the same for two consecutive scans, the host machine's state is switched from a synchronization state to a stable state, and a first notification is sent to the server, indicating that the data written by the virtual machine to the host machine's storage space has been sent to the server.

[0070] The scanning process involves scanning from the initial address to the final address of the storage space. It should be understood that scanning from the initial address to the final address of the storage space constitutes one scan cycle.

[0071] In some embodiments, after the first scan is completed, it is checked whether there are still "dirty" bits in the bitmap. If so, an incremental scan is performed on these "dirty" bits, only rereading and sending the data corresponding to the "dirty" bits to the server, and then clearing the corresponding bitmap bits again, marking them as backed up to the server. This process is repeated iteratively, with the number of "dirty" bits to be scanned decreasing monotonically each time, until the number of remaining "dirty" bits drops below a preset threshold or no new "dirty" bits are added between two consecutive scans.

[0072] In practical applications, when a virtual machine starts up, it needs to synchronize the data in the storage space and the data newly written by write requests to the server. At this time, the host machine is in a synchronized state. When the number of subspaces that are not backed up to the server is less than a preset threshold or the increment of the number is the same twice consecutively, the host machine switches from the synchronized state to a stable state, and subsequently only responds to write requests to synchronize the written data.

[0073] The first notification carries the sequence number of the data update log at the current moment, which indicates that the log corresponding to the sequence number and the previous logs have been sent to the server, and that the data written by the virtual machine to the host machine's storage space has been sent to the server, so that the server can persist the received data into a complete basic disk image.

[0074] According to embodiments of this disclosure, the increment of the number of unbacked-up subspaces obtained from each scan is acquired to understand the changing trend of unbacked-up data. If the number of unbacked-up subspaces is less than a preset threshold or if two consecutive increments are the same, the host machine state is switched from synchronization state to stable state, indicating that full data synchronization is nearing completion. A first notification is then sent to the server, indicating that the data in the storage space has been sent to the server, so that the server can confirm that full synchronization is complete.

[0075] In some exemplary embodiments, the data update log generation method further includes: when the host machine switches to a stable state, if the cumulative duration from the last time multiple data update logs were packaged and sent to the current time is greater than or equal to a preset threshold, or the number of multiple data update logs is greater than or equal to a number threshold, the multiple data update logs are packaged and sent to the server.

[0076] The preset threshold represents the cumulative duration and is used to measure how often a data update log is packaged and sent.

[0077] In practical applications, a preset threshold is used as a configurable time window, such as sending a data update log every 5ms, or accumulating the number of data update logs until a threshold is reached, such as sending a data update log every 4MB, in order to reduce network round-trip latency and protocol header overhead.

[0078] According to embodiments of this disclosure, a stable host state indicates that the data in the storage space has been fully synchronized, and attention only needs to be paid to data updates caused by write requests. Therefore, logs need to be packaged and sent subsequently to synchronize the server. The timing for triggering packaged sending is accurately determined by whether the cumulative time since the last packaged sending of data update logs reaches a preset threshold or whether the number of logs reaches a certain threshold. When the conditions are met, multiple data update logs are packaged and sent to the server, achieving incremental backup in a stable state and avoiding the overhead of frequent transmissions.

[0079] In some exemplary embodiments, the data update log generation method further includes storing multiple data update logs in a first preset buffer on the host machine; if the data volume of the multiple data update logs exceeds the capacity of the first preset buffer, storing the data update logs overflowing the first preset buffer in a second preset buffer on the host machine. If the data volume of the multiple data update logs exceeds a capacity threshold, an alarm message is generated.

[0080] Among them, the data update logs in the second preset buffer are sent to the server with a higher priority than the data update logs in the first preset buffer.

[0081] The capacity threshold is determined based on the capacity of the first preset buffer and the capacity of the second preset buffer.

[0082] In practical applications, when the data volume of multiple data update logs approaches the capacity of the first preset buffer, if the host machine is in a synchronized state, it means that the data written by the virtual machine to the host machine's storage space has not been fully synchronized to the server and is still in a full synchronization state. At this time, the data transmission link between the host machine and the server is congested. Therefore, it is necessary to reduce the I / O bandwidth quota of the full synchronization coroutine and give more buffer space to the data update log, that is, reduce the full synchronization rate and increase the synchronization rate of the data update log.

[0083] When reducing the rate of full synchronization still fails to prevent the data volume of the data update log from continuing to rise and approaching the capacity of the first preset buffer, the overflowing data update log is temporarily written to the second preset buffer. The second preset buffer can be an overflow log file on the host's local disk, which uses append-only writing mode with order preservation as the first priority. After the data transmission link between the host and the server returns to a normal state from a congested state, the worker thread prioritizes reading and resuming the transmission of the data update log from the second preset buffer, and deletes the file after the transmission is complete.

[0084] If the amount of data in multiple data update logs exceeds the capacity threshold, an alarm message for Continuous Data Protection (CDP) protection degradation will be sent to notify the administrator to intervene.

[0085] According to embodiments of this disclosure, data update logs are stored in a first preset buffer on the host machine for temporary log management. When the log data volume exceeds the capacity of the first preset buffer, the overflowing logs are stored in a second preset buffer, and the log transmission priority of the second preset buffer is higher than that of the first preset buffer, thereby ensuring priority transmission of overflowing data. When the log data volume exceeds a capacity threshold determined by the capacities of the two buffers, an alarm message is generated to promptly alert the risk of insufficient storage resources.

[0086] In some exemplary embodiments, the data update log generation method further includes: in response to receiving a disk flushing instruction or a forced unit access instruction from a virtual machine, sending a second notification to the server, indicating that the payload data of the write request sent from the virtual machine startup to the current time has been stored to the physical hard disk.

[0087] In practical applications, the Flush command instructs the writing of all data in the cache to the physical hard drive, while the Forced Unit Access (FUA) command instructs a forced access to the physical hard drive. Receiving either a Flush or Forced Unit Access command indicates that the payload data for all previous write requests has been confirmed by the virtual machine's guest operating system to be stored on the physical hard drive. When performing data recovery, the time point marked by the second notification can be selected for data recovery.

[0088] According to embodiments of this disclosure, in response to a disk flushing command or a forced unit access command from a virtual machine, a second notification is sent to the server, indicating that the write request payload data from the virtual machine startup to the current time has been stored on the physical hard disk, so that the server knows that the time point corresponding to the second notification can be used as a recoverable time point and thus recover the data before that time point.

[0089] This disclosure also provides a data backup method applied to a server, the data backup method including backing up the data update log in response to receiving a data update log from the host machine, wherein the data update log is generated using a data update log generation method.

[0090] Figure 3 A flowchart illustrating a data update log generation method according to another embodiment of this disclosure is shown schematically.

[0091] like Figure 3 As shown, the data update log generation method in this embodiment includes operations S301 to S304, which correspond to the steps of generating a data update log after the host machine receives a write request.

[0092] In operation S301, a write request was received from the virtual machine.

[0093] In practical applications, the virtual machine sends a write request to the host machine. Specifically, the virtual machine's Guest OS sends the write request, which is then obtained by the host machine through the Guest OS's file system, the Guest OS's block device driver layer, and the host machine's VirtIO block device.

[0094] When operating S302, the host machine's filter intercepts the write request.

[0095] It should be understood that the host machine's filter is... Figure 1 The filter in the embodiment.

[0096] When operating S303, the host machine's filter forwards write requests to the host machine's virtual disk.

[0097] It should be understood that forwarding the original write request to the virtual disk without any modification ensures that the payload data of the write request is stored in the target subspace indicated by the write request, thus preventing the virtual machine's normal business I / O from being blocked and minimizing the impact on virtual machine I / O performance. The host machine's virtual disk is... Figure 1 The virtual disk in the embodiment.

[0098] In practical applications, after forwarding the write request to the host machine's virtual disk, the status is returned to the virtual machine's Guest OS.

[0099] When operating S304, the host machine's filter generates data update logs based on write requests.

[0100] In practical applications, operation S304 is performed simultaneously with operation S303, or operation S304 is performed immediately after operation S303 is completed.

[0101] It should be understood that the host machine's filter parses the write request to obtain the data offset, data length, and payload data, and combines them with the sequence number that represents the data update log generation time to generate the data update log, and stores the data update log in the first preset buffer.

[0102] In practical applications, the filter writes the data update log to a first preset buffer that is pre-allocated in the host machine's memory.

[0103] The first preset buffer stores and forwards data update logs using either a single producer-single consumer or multiple producer-single consumer models. Specifically, the producer is an I / O coroutine of the Quick EMUlator (QEMU), and the consumer is a background asynchronous transmission worker thread. An atomic operation-based lock-free enqueue algorithm is employed to avoid the context switching and priority inversion overhead of traditional mutex locks. The first preset buffer allocates a fixed-size contiguous memory region (e.g., 64MB~256MB) during initialization, avoiding runtime dynamic memory allocation overhead. The producer cursor (Head) and consumer cursor (Tail) are aligned to independent CPU cache lines, eliminating cache coherency protocol overhead caused by false sharing. When the host receives an acknowledgment sequence number from the server (the maximum sequence number received by the server), the consumer cursor is advanced based on the acknowledgment sequence number, releasing the acknowledged buffer space. If no acknowledgment sequence number is received within the timeout period, a retransmission mechanism is triggered, and retransmission begins from the unacknowledged sequence number position.

[0104] As can be seen, the first preset buffer decouples the generation and transmission of data update logs. Write request interception only needs to generate the data update log once, while transmission is completed asynchronously by a background thread, thus avoiding network latency jitter from affecting virtual machine performance. Furthermore, when transmitting data update logs, the packet size and sending frequency are dynamically adjusted based on real-time network bandwidth utilization and the capacity of the first preset buffer.

[0105] When the server receives each data update log, it verifies whether the sequence number differs from the sequence number of the previously received data update log by 1. If a sequence number jump is detected, a log break notification is immediately sent to the host machine's filter, carrying the missing sequence number range. If the data update log chain is broken due to abnormal situations such as network interruption, server failure, or overflow of the first preset buffer, and the log chain break is irreparable (i.e., the data update logs corresponding to some sequence numbers are lost and cannot be recovered from the second preset buffer), all data marked as not synchronized to the server from the previous confirmed sequence number to the current sequence number in the bitmap is extracted to form a bitmap to be resynchronized. Using this bitmap to be resynchronized as the scanning range, data is read and retransmitted to the server. This ensures that the data transfer volume during the recovery process is only proportional to the amount of data actually written during the break, rather than the total virtual disk capacity, significantly shortening the recovery time.

[0106] After resynchronization is complete, the filter sends a resynchronization completion flag to the server, carrying the sequence number range covered by the resynchronization and the latest sequence number. The server applies the resynchronized data to the base image and re-establishes the incremental log chain starting from the sequence number corresponding to the flag.

[0107] Figure 4 A flowchart illustrating a data update log generation method according to another embodiment of this disclosure is shown schematically.

[0108] like Figure 4 As shown, this embodiment includes operations S401 to S404.

[0109] In operation S401, initialize the bitmap corresponding to the virtual disk capacity.

[0110] In operation S402, the data of each subspace is sequentially determined to be backed up to the server based on the bitmap.

[0111] In operation S403, in response to receiving a write request, the bitmap of the target subspace storing the payload data of the write request is determined to be not backed up to the server.

[0112] When operating S404, if the number of subspaces whose backup status is not backed up to the server is less than a preset threshold or the number increments are the same twice consecutively, the full scan is stopped, and data update logs are generated and sent to the server only based on the received write requests.

[0113] It should be understood that stopping the full scan means that the data written by the virtual machine to the host machine's storage space has been sent to the server. Subsequent updates only require updating the data upon receiving write requests, i.e., sending the corresponding data update log to the server. Before stopping the full scan, the host machine is in a state of full synchronization, meaning that not all the data written by the virtual machine to the host machine's storage space has been sent to the server. Therefore, the full scan and write request interception run in parallel.

[0114] Figure 5 A flowchart illustrating a host state transition process according to an embodiment of the present disclosure is shown schematically.

[0115] like Figure 5 As shown, this embodiment includes operations S501 to S503.

[0116] When operating S501, the host machine is in its initial state.

[0117] When operating S502, the host machine is in a synchronization state.

[0118] It should be understood that, in response to virtual machine startup, a filter is dynamically inserted into the block device chain of the virtual disk corresponding to the virtual machine within the QEMU process, and the host machine switches from the initial state to the synchronous state. In some embodiments, if the virtual machine has already started and the administrator needs to synchronize the virtual machine's data without inserting a filter, a filter is inserted via hot-swapping, and the host machine switches from the initial state to the synchronous state.

[0119] When operating S503, the host machine is in a stable state.

[0120] Figure 6 A schematic block diagram of an apparatus according to an embodiment of the present disclosure is shown.

[0121] like Figure 6 As shown, the apparatus 600 of this embodiment includes...

[0122] The backup status acquisition module 610 is used to acquire the backup status of data written by the virtual machine to the host machine's storage space in response to the virtual machine startup. The storage space includes multiple sub-spaces, and the backup status includes whether it has been backed up to the server or not. In one embodiment, the backup status acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0123] The target subspace determination module 620 is configured to, in response to receiving at least one write request from a virtual machine, determine, for each write request, a target subspace from a plurality of subspaces for storing payload data carried by the write request, and store the payload data in the target subspace. In one embodiment, the target subspace determination module 620 may be used to perform the operation S220 described above, which will not be repeated here.

[0124] The data update log generation module 630 is used to determine that the backup status of the payload data in the target subspace is not backed up to the server, and to generate a data update log for the target subspace based on write requests. The data update log is used to characterize the data update status of the target subspace. In one embodiment, the data update log generation module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0125] The log sending module 640 is used to package and send the data update logs of multiple write requests to the server so that the server can back up the data update logs. In one embodiment, the log sending module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0126] This disclosure also provides a data backup device, including: a log backup module, configured to back up the data update log in response to receiving a data update log from a host machine, wherein the data update log is generated using a data update log generation method.

[0127] Figure 7 A schematic block diagram of an electronic device that can be used to implement the methods of embodiments of the present disclosure is shown.

[0128] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0129] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also implement the methods provided in embodiments of this disclosure by executing programs stored in one or more memories.

[0130] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0131] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0132] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0133] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0134] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0135] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0136] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0137] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information in this disclosed technical solution comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. In this disclosed technical solution, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0138] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0141] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for generating data update logs, applied to a host machine, characterized in that, The method includes: In response to the virtual machine startup, the backup status of the data written by the virtual machine to the host machine's storage space is obtained. The storage space includes multiple sub-spaces, and the backup status includes whether it has been backed up to the server or not. In response to receiving at least one write request from the virtual machine, for each write request, a target subspace for storing the payload data carried by the write request is determined from the plurality of subspaces based on the write request, and the payload data is stored in the target subspace; The backup status of the payload data in the target subspace is determined to be not backed up to the server, and a data update log for the target subspace is generated based on the write request. The data update log is used to characterize the data update status of the target subspace. The data update logs of multiple write requests are packaged and sent to the server so that the server can back up the data update logs.

2. The method according to claim 1, characterized in that, The method further includes: If the data in the subspace is not backed up to the server, obtain the starting address of the multiple subspaces; Based on the starting address, the data of the multiple subspaces are sequentially sent to the server, and the backup status of the data is updated to "backed up to the server".

3. The method according to claim 1 or 2, characterized in that, The step of generating the data update log for the target subspace based on the write request includes: Based on the write request, a sequence number is determined to characterize the generation time of the data update log; The data update log is generated based on the sequence number and the data offset, data length, and payload data obtained by parsing the write request. The data offset indicates the starting address for storing the payload data in the storage space.

4. The method according to claim 1, characterized in that, The method further includes: The increment of the number of subspaces not backed up to the server is obtained for each scan, wherein the scan includes scanning from the initial address to the end address of the storage space; If the number of subspaces whose backup status is not backed up to the server is less than a preset threshold or if the increment of the number is the same twice consecutively, the host machine's state is switched from synchronization state to stable state. A first notification is sent to the server, indicating that the data written by the virtual machine to the host machine's storage space has been sent to the server.

5. The method according to claim 4, characterized in that, The method further includes: When the host machine switches to the stable state, if the cumulative duration from the last time multiple data update logs were packaged and sent to the current time is greater than or equal to a preset threshold, or if the number of multiple data update logs is greater than or equal to a number threshold, the multiple data update logs are packaged and sent to the server.

6. The method according to claim 1, characterized in that, The method further includes: The data update logs are stored in the first preset buffer of the host machine; If the amount of data in multiple data update logs exceeds the capacity of the first preset buffer, the multiple data update logs that overflow the first preset buffer are stored in the second preset buffer of the host machine. The data update logs in the second preset buffer are sent to the server with a higher priority than the data update logs in the first preset buffer. If the amount of data in multiple data update logs exceeds a capacity threshold, an alarm message is generated. The capacity threshold is determined based on the capacity of the first preset buffer and the capacity of the second preset buffer.

7. The method according to claim 1, characterized in that, The method further includes: In response to receiving a disk flush command or forced unit access command from the virtual machine, a second notification is sent to the server, indicating that the payload data of the write request sent by the virtual machine from startup to the current time has been stored to the physical hard disk.

8. A data backup method, applied to a server, characterized in that, The method includes: In response to receiving a data update log from the host machine, the data update log is backed up, wherein the data update log is generated using the method according to any one of claims 1 to 7.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images