Method, system and device for reducing space occupied by snapshot device and medium
By using character devices and block devices to simulate snapshot devices in the server, only writing-on-time copy operations are performed on the data blocks that need to be backed up, the problem of excessive storage space for snapshot devices is solved, and more efficient data backup and recovery is achieved.
Patent Information
- Application Number
- CN202411145587.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-13
AI Technical Summary
In the prior art, when using the write-on-write method for data backup, the snapshot device occupies too much storage space, resulting in the problem of performance degradation and snapshot failure when data write operations are frequent on the server.
By adding character devices and block devices to the server, the block device takes over I/O requests of the logical volume, monitors disk data changes in real time, performs write-on-time copy operations on data blocks that need to be backed up, and uses a balanced binary tree structure to store COW data, avoid processing of all changing data.
Reduce the storage space occupied by snapshot devices, avoid snapshot creation failures caused by large COW data, and improve backup efficiency and performance.
Smart Images
Figure CN119127069A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data backup, and in particular to a method, system, device and medium for reducing space occupied by a snapshot device. Background Art
[0002] Data backup is the basis of disaster recovery. In order to prevent data loss due to system failures and other reasons, backing up the data on the host is an essential operation. A common data backup method is to use snapshots to back up the system. When recovery is needed, the backed-up snapshot is used to restore the data. Common technologies for creating snapshots include copy-on-write (COW) and redirect-on-write (ROW). When using the COW method to create snapshots, the advantage is that it takes a short time and is fast to create snapshots; the disadvantage is that the COW size needs to be specified when creating a snapshot. When the data write operations on the server are frequent and large, the machine write performance will be reduced, and the storage space required for the snapshot COW will increase. This will result in insufficient space reserved for the snapshot, making the snapshot invalid and unable to restore the data. Summary of the invention
[0003] The purpose of the present invention is to provide a method, system, device and medium for reducing the space occupied by snapshot devices, optimize the copy-on-write technology, and reduce the storage space occupied by snapshot devices during the backup process.
[0004] To achieve the above purpose, the technical solution provided by the present invention is:
[0005] A first aspect of the present application provides a method for reducing the space occupied by snapshot devices, using character devices and block devices to simulate snapshot devices, including: when backing up data on a server, simulating snapshot device creation by adding a character device in the server; at the same time, adding a block device in the general layer based on the logical volume information on the server, and when data backup and disk data reading are required, data is read by accessing the block device, and the data blocks that need to be backed up are determined through the block device.
[0006] To optimize the above technical solutions, the specific measures taken also include:
[0007] The block device takes over all I / O requests on the logical volume; after receiving the I / O request, the block device returns the disk backup data to the upper layer application, and monitors the changes of the disk data on the server in real time to form copy-on-write (COW) data.
[0008] When reading disk data, the block device obtains bitmap information of the disk, records data blocks containing data through the bitmap information, and monitors data changes of the data blocks containing data.
[0009] When disk data changes, an I / O request for the changed data is sent to the block device, which marks the changed data of the existing data that has not been backed up as write-time copy data in the initial disk bitmap, and stores it after performing a write-time copy operation on it.
[0010] Preferably, the block device adopts a balanced binary tree data structure in the copy-on-write operation on data.
[0011] Furthermore, the method for reducing the space occupied by the snapshot device specifically comprises the following steps:
[0012] Start the backup program on the server;
[0013] Install character devices on the server;
[0014] The backup program uses logical volumes as units and notifies the kernel to add block devices in the general layer based on the logical volume information.
[0015] The block device obtains disk bitmap information and determines the data blocks that need to be backed up;
[0016] The backup program reads the backup data;
[0017] The backup program stores the backup data.
[0018] Furthermore, the block device receives an I / O request on the original logical volume:
[0019] When it is a write request, determine whether there is data on the original data block. If there is no data, directly send the I / O request to the disk; when it is a write request and there is data on the original data block, but the data on the data block has been backed up, directly send the I / O request to the disk; when it is a write request, there is data on the original data block, and the data has not been backed up, perform a copy-on-write operation on the original data block, put its data as a leaf node on the balanced binary tree, rebalance the balanced binary tree, and then the block device sends the I / O request to the disk;
[0020] When it is a read request, if the data on the read data block has not been copied on write, the request is directly sent to the disk; when it is a read request, if the data on the read data block has been copied on write, the corresponding data is searched on the balanced binary tree, and after the data is returned to the upper-level application, the data is cleared from the tree and the balanced binary tree is rebalanced.
[0021] A second aspect of the present application provides a system for reducing the space occupied by a snapshot device, comprising:
[0022] The character device installation module is used to install character devices on the server to simulate the creation of snapshot devices;
[0023] The block device adding module is used to add block devices in the general layer according to the logical volume information on the server; the block device is used to obtain disk bitmap information, record data changes in the disk, and determine the data blocks that need to be backed up through the block device;
[0024] The backup module is used to read the backup data and store the backup data.
[0025] A third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for reducing the space occupied by a snapshot device as described above is implemented.
[0026] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for reducing the space occupied by a snapshot device as described above.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] When performing data backup, the present invention no longer creates a snapshot device, but adds a character device in the memory, and adds a block device on the general layer to simulate the snapshot device, thereby reducing the storage space occupied by the snapshot device; the block device takes over the I / O request of the original logical volume, obtains the bitmap information of the original disk, and only monitors the data blocks marked with data.
[0029] The present invention can reduce the storage space occupied by the snapshot device during the backup process, especially when a large amount of copy-on-write (COW) data is generated, the snapshot reserved space is occupied. The present invention avoids the problem of snapshot creation failure caused by the large amount of COW data, thereby causing backup failure, by reducing the reserved space occupied by COW data during the backup process.
[0030] The present invention receives disk I / O requests through block devices, filters the I / O requests, performs copy-on-write (COW) processing on data on data blocks that have changed and need to be backed up but have not been backed up in time, and stores them in a balanced binary tree. When this data is backed up, the data is returned to an upper-layer application, and the data is deleted from the balanced binary tree. In the entire process, no snapshot device is created, thereby reducing the storage space occupied by the snapshot device. COW processing is not performed on all changed data, and only data that really needs COW is stored, thereby reducing the storage space occupied by COW data. The character device continues to run in the memory, and when the next backup is performed, incremental data can be easily obtained through bitmap information, and incremental data backup is directly performed, thereby improving backup efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 : A specific flow chart of the method for reducing the space occupied by snapshot devices of the present invention.
[0032] Figure 2 : A flow chart of a method for determining data blocks that need to be backed up and performing COW processing in the present invention.
[0033] Figure 3 : Schematic diagram of the method flow of reading backup data of the present invention. DETAILED DESCRIPTION
[0034] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.
[0035] Some of the terms in the present invention are explained as follows:
[0036] COW (Copy-On-Write), also known as "copy-on-write" snapshot technology or "copy-on-write" snapshot technology, is also commonly referred to as "metadata (source data pointer table)" copy.
[0037] Snapshot is a data copy technology based on a time point. Its purpose is to record data information at a certain moment and save it. If some failure occurs later and data recovery is required, the data can be restored to the state at the previous time point through the snapshot.
[0038] In one embodiment, the present invention proposes a method for reducing the space occupied by snapshot devices, using character devices and block devices to simulate snapshot devices, including: when backing up data on a server, simulating the creation of a snapshot device by adding a character device in the server; at the same time, adding a block device in the general layer based on the logical volume information on the server, when data backup and disk data reading are required, data are read by accessing the block device, and the data blocks that need to be backed up are determined through the block device.
[0039] The block device takes over all I / O requests on the logical volume; after receiving the I / O request, the block device returns the disk backup data to the upper-layer application, and monitors the changes of disk data on the server in real time to form write-time copy data.
[0040] When reading disk data, the block device obtains the bitmap information of the disk, records the data blocks containing data through the bitmap information, and monitors the data changes of the data blocks containing data.
[0041] When disk data changes, an I / O request for the changed data is sent to the block device. The block device marks the changed data of the existing data that has not been backed up as write-time copy data in the initial disk bitmap, and stores it after performing a write-time copy operation on it.
[0042] In a disclosed embodiment of the present invention, Figure 1 As shown, the method for reducing the space occupied by the snapshot device specifically includes the following steps:
[0043] Start the backup program backupProcess on the server;
[0044] Install the character device / dev / charDev on the server;
[0045] The backup program backupProcess uses the logical volume / dev / sda1 as a unit and notifies the kernel to add a block device / dev / sda1BlockDev in the general layer based on the logical volume information.
[0046] The block device / dev / sda1BlockDev obtains the disk bitmap information and determines the data blocks that need to be backed up;
[0047] The backup program reads the backup data;
[0048] The backup program stores the backup data.
[0049] When the block device / dev / sda1BlockDev receives the I / O request on the original logical volume / dev / sdal:
[0050] like Figure 2 As shown, when it is a write request, it is determined whether there is data on the original data block. If there is no data, the I / O request is directly sent to the disk; when it is a write request and there is data on the original data block, but the data on the data block has completed the backup operation, the I / O request is directly sent to the disk; when it is a write request, there is data on the original data block, and the data has not been backed up, it is necessary to perform a COW operation on the original data, put the data as a leaf node on the balanced binary tree, rebalance the balanced binary tree, and then the block device sends the I / O request to the disk;
[0051] like Figure 3 As shown, when it is a read request, if the data of the read data block has not undergone the COW operation, the request is directly sent to the disk; when it is a read request, if the data of the read data block has undergone the COW operation, the data is searched on the balanced binary tree, and after the data is returned to the upper-level application backupProcess, the data is cleared from the tree and the balanced binary tree is balanced again.
[0052] From then on, the backup program runs, character devices are installed, block devices are added, disk bitmap information is obtained, block devices receive disk I / O requests, I / O requests are filtered, and COW processing is performed on the block data that has changed and needs to be backed up, but has not been backed up in time, and the data on the data blocks is stored in a balanced binary tree. When this data is backed up, the data is returned to the upper-level application and deleted from the balanced binary tree. In the whole process, no snapshot device is created, which reduces the storage space occupied by the snapshot device; COW processing is not performed on all changed data, and only the data that really needs COW is stored, which reduces the storage space occupied by COW data.
[0053] The present invention uses a character device and a block device to simulate a snapshot device. When backing up data on a server, a snapshot device is first created. At this time, a real snapshot device is no longer created for the machine, but a character device is added to the server. The character device is loaded and runs in the memory and does not occupy the storage space of the server. At the same time, according to the logical volume information on the server, a block device is added to the general layer, and the block device takes over all I / O requests on the logical volume. When data backup is performed to read disk data, the application accesses the block device to read the data. After the block device receives the I / O request, it returns the disk backup data to the upper-layer application, and monitors the changes of the disk data on the server in real time. This part of the changed data is COW data. The block device will obtain the bitmap information of the disk, and the bitmap information will record which data blocks have data and which data blocks do not have data. Only the data blocks with data will be monitored for data changes. When disk data changes, the I / O request for the changed data is sent to the block device. At this time, the block device will not perform COW data processing for all changes. When the changed data block has been backed up, the block device will directly send the I / O request for this part of the data to the disk without COW data processing. When the changed data block is marked as having no data in the initial bitmap, it is a newly added data, and COW data processing is not performed at this time. Only when there is data before the changed data block and it has not been backed up, this part of the data will be stored as COW data. When storing, the data structure of the balanced binary tree is used to improve query efficiency. When the COW data is backed up, the data will be deleted from the balanced binary tree, and the balanced binary tree will be rebalanced. As the backup progresses, COW data is continuously read and storage space is released. When the logical volume completes the data backup, the corresponding block device is removed from the general layer.
[0054] During the backup process, no real snapshot device is created. Only a character device is added to the memory and a block device is added to the general layer to simulate the snapshot. These two devices do not occupy storage space. During the backup process, data changes can be monitored in real time. Not all changed data will be processed as COW data. Only the block data that has not been backed up and has had data on it but has changed will be processed as COW data, reducing the storage space occupied by COW data. When processing COW data, the balanced binary tree data structure is used to improve query efficiency. When COW data is read, it will be cleared immediately, reducing the storage space occupied by COW data during the backup process.
[0055] In another embodiment, the present invention provides a system for reducing the space occupied by a snapshot device, comprising:
[0056] The character device installation module is used to install character devices on the server to simulate the creation of snapshot devices;
[0057] The block device adding module is used to add block devices in the general layer according to the logical volume information on the server; the block device is used to obtain disk bitmap information, record data changes in the disk, and determine the data blocks that need to be backed up through the block device;
[0058] The backup module is used to read the backup data and store the backup data.
[0059] In another embodiment, the present invention proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method of reducing the space occupied by the snapshot device is implemented as described above.
[0060] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above method for reducing the space occupied by a snapshot device.
[0061] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0062] When performing data backup, the present invention no longer creates a snapshot device, but adds a character device in the memory, and adds a block device on the general layer to simulate the snapshot device, thereby reducing the occupation of storage space by the snapshot device.
[0063] The block device of the present invention takes over the I / O request of the original logical volume, obtains the bitmap information of the original disk, and only monitors the data blocks marked with data.
[0064] The present invention performs filtering operation on the changed data, and does not perform COW operation on all the changed data. The COW operation is performed only when there is no backup and the data of the data block part marked as having data on the original bitmap changes.
[0065] The present invention uses a balanced binary tree to store COW data, thereby improving data query efficiency. After the data is read, it is deleted from the balanced binary tree, thereby saving storage space.
[0066] The character device of the present invention runs continuously in the memory. When the next backup is performed, the incremental data can be easily obtained through the bitmap information, and the incremental data can be directly backed up, thereby improving the backup efficiency.
[0067] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention and based on the technical essence of the present invention, may make any simple modifications, equivalent substitutions and improvements to the above embodiments, which are still within the protection scope of the technical solution of the present invention.
Claims
1. A method for reducing the space occupied by a snapshot device, characterized in that: Use character devices and block devices to simulate snapshot devices, including: when backing up data on a server, simulate snapshot device creation by adding a character device in the server; at the same time, add a block device in the general layer based on the logical volume information on the server, and when data backup and disk data reading are required, read the data by accessing the block device, and determine the data blocks that need to be backed up through the block device.
2. The method for reducing the space occupied by snapshot devices according to claim 1, characterized in that: The block device takes over all I / O requests on the logical volume; after receiving the I / O request, the block device returns the disk backup data to the upper layer application, and monitors the changes of the disk data on the server in real time to form write-time copy data.
3. The method for reducing the space occupied by snapshot devices according to claim 1, characterized in that: When reading disk data, the block device obtains the bitmap information of the disk, records the data blocks containing data through the bitmap information, and monitors the data changes of the data blocks containing data.
4. The method for reducing the space occupied by snapshot devices according to claim 3, characterized in that: When disk data changes, an I / O request for the changed data is sent to the block device, which marks the changed data of the existing data that has not been backed up as write-time copy data in the initial disk bitmap, and stores it after performing a write-time copy operation on it.
5. The method for reducing the space occupied by snapshot devices according to claim 4, characterized in that: The block device uses a balanced binary tree data structure in the copy-on-write operation on data.
6. The method for reducing the space occupied by snapshot devices according to claim 1, characterized in that: The specific steps include: Start the backup program on the server; Install character devices on the server; The backup program uses logical volumes as units and notifies the kernel to add block devices in the general layer based on the logical volume information. The block device obtains disk bitmap information and determines the data blocks that need to be backed up; The backup program reads the backup data; The backup program stores the backup data.
7. The method for reducing the space occupied by snapshot devices according to claim 1, characterized in that: The block device receives the I / O request on the original logical volume: When it is a write request, determine whether there is data on the original data block. If there is no data, directly send the I / O request to the disk; when it is a write request and there is data on the original data block, but the data on the data block has been backed up, directly send the I / O request to the disk; when it is a write request, there is data on the original data block, and the data has not been backed up, perform a copy-on-write operation on the original data block, put its data as a leaf node on the balanced binary tree, rebalance the balanced binary tree, and then the block device sends the I / O request to the disk; When it is a read request, if the data on the read data block has not been copied on write, the request is directly sent to the disk; when it is a read request, if the data on the read data block has been copied on write, the corresponding data is searched on the balanced binary tree, and after the data is returned to the upper-level application, the data is cleared from the tree and the balanced binary tree is rebalanced.
8. A system for reducing the space occupied by a snapshot device, characterized in that: include: The character device installation module is used to install character devices on the server to simulate the creation of snapshot devices; The block device adding module is used to add block devices in the general layer according to the logical volume information on the server; The block device is used to obtain disk bitmap information, record data changes in the disk, and determine the data blocks that need to be backed up through the block device; The backup module is used to read the backup data and store the backup data.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for reducing the space occupied by a snapshot device as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for reducing the space occupied by a snapshot device according to any one of claims 1 to 7.