Snapshot data processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a snapshot data processing method, device, electronic device, and storage medium, which belong to the field of data storage. The method includes: obtaining multiple snapshots; wherein the multiple snapshots are arranged according to the generation time to form a snapshot chain, each snapshot includes multiple data blocks, and the data blocks include empty data blocks and non-empty data blocks; among the multiple snapshots in the snapshot chain, determining the target flattened snapshot; for each empty data block in the target flattened snapshot, searching in sequence in the snapshot chain along the direction of the generation time earlier than the target flattened snapshot to determine the target data block corresponding to each empty data block; obtaining the storage information in each target data block, and filling the empty data block corresponding to each target data block according to the storage information. The embodiments of the present application can improve the reading efficiency of snapshots.

Description

Technical Field

[0001] The present application relates to the field of data storage, and in particular to a snapshot data processing method, device, electronic device, and storage medium. Background Art

[0002] In modern distributed storage systems, snapshot technology is widely used to meet data protection and management requirements. One mainstream implementation is ROW (Redact-On-Write) snapshot technology. ROW snapshots allow users to create instantaneous copies of data without disturbing the original data. This is crucial for scenarios such as data backup, recovery, and multi-version data management.

[0003] In related technologies, ROW snapshot technology still has drawbacks in data read performance. When reading data, if the required data is not in the current snapshot, the system must traverse the snapshot chain until the required data is found. This process can result in significant performance overhead, especially when there are a large number of snapshots and frequent data updates. Furthermore, since each read may require traversing multiple snapshots, read performance is highly volatile. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide a snapshot data processing method, device, electronic device and storage medium, which can improve the reading efficiency of snapshots.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a snapshot data processing method, the method comprising:

[0006] Acquire multiple snapshots; wherein the multiple snapshots are arranged according to the generation time to form a snapshot chain, each of the snapshots includes multiple data blocks, and the data blocks include empty data blocks and non-empty data blocks;

[0007] Determining a target flattened snapshot among the plurality of snapshots in the snapshot chain;

[0008] For each empty data block in the target flattened snapshot, sequentially search the snapshot chain along a direction whose generation time is earlier than the target flattened snapshot to determine a target data block corresponding to each empty data block; wherein the target data block is a non-empty data block;

[0009] The storage information in each of the target data blocks is acquired, and the empty data blocks corresponding to each of the target data blocks are filled according to the storage information.

[0010] In some embodiments, for each empty data block in the target flattened snapshot, sequentially searching the snapshot chain along a direction where the generation time is earlier than the target flattened snapshot to determine the target data block corresponding to each empty data block includes:

[0011] Determining a data block sequence number of each of the empty data blocks in the target flattened snapshot;

[0012] According to the data block sequence number and the snapshot chain, the data blocks with the same data block sequence number as the empty data block in each snapshot are retrieved in order of generation time from earliest to latest.

[0013] In response to retrieving the first non-empty data block, the non-empty data block is determined as the target data block corresponding to the empty data block.

[0014] In some embodiments, obtaining storage information in each of the target data blocks and filling the empty data blocks corresponding to each of the target data blocks according to the storage information includes:

[0015] Acquire the storage information in each target data block; wherein the storage information includes a data storage address;

[0016] The data storage address is copied to the storage information of the empty data block corresponding to the target data block.

[0017] In some embodiments, the storage information further includes a data length and a data check code, and before copying the data storage address to the empty data block corresponding to the target data block, the process further includes:

[0018] Read the corresponding data content according to the data storage address;

[0019] Verify the read data content using the data length and the data check code;

[0020] The data length and the data check code are copied to the storage information of the empty data block.

[0021] In some embodiments, before sequentially searching the snapshot chain along a direction where the generation time of each empty data block in the target flattened snapshot is earlier than the target flattened snapshot and determining the target data block corresponding to each empty data block, the method further includes:

[0022] Obtaining a preset snapshot leveling interval and a snapshot sequence number of the target leveling snapshot;

[0023] Determining a snapshot span between the target leveled snapshot and the snapshot sequence number and the snapshot leveling interval in the snapshot chain, and determining the snapshots within the snapshot span as a target search range;

[0024] The step of sequentially searching the snapshot chain along a direction where the snapshot is generated earlier than the target flattened snapshot to determine the target data block corresponding to each empty data block includes:

[0025] Within the target search range, the snapshot chain is searched sequentially along a direction where a snapshot generated earlier than the target flattened snapshot is generated, to determine a target data block corresponding to each empty data block.

[0026] In some embodiments, before determining a target leveled snapshot among the plurality of snapshots in the snapshot chain, the method further includes:

[0027] Get the preset flattening operation time period and get the current time in real time;

[0028] When the current time is in the flattening operation time period, continue to execute:

[0029] A target flattened snapshot is determined among the plurality of snapshots in the snapshot chain.

[0030] In some embodiments, determining a target leveled snapshot among the plurality of snapshots in the snapshot chain includes:

[0031] Get the preset non-empty data block ratio threshold;

[0032] For each snapshot in the snapshot chain, obtain the number of non-empty data blocks in the snapshot and a preset total number of data blocks;

[0033] Calculating the ratio of the number of non-empty data blocks to the total number of data blocks to obtain a non-empty data block ratio of the snapshot;

[0034] When the non-empty data block ratio of the snapshot is less than or equal to the non-empty data block ratio threshold, the snapshot is determined as the target flattened snapshot.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a snapshot data processing device, the device comprising:

[0036] A first acquisition module is configured to acquire a plurality of snapshots, wherein the plurality of snapshots are arranged according to generation time to form a snapshot chain, each of the snapshots comprising a plurality of data blocks, wherein the data blocks comprise empty data blocks and non-empty data blocks;

[0037] a determination module, configured to determine a target leveled snapshot among the plurality of snapshots in the snapshot chain;

[0038] a retrieval module configured to sequentially search, for each empty data block in the target flattened snapshot, the snapshot chain along a direction where the generation time is earlier than the target flattened snapshot, to determine a target data block corresponding to each empty data block; wherein the target data block is a non-empty data block;

[0039] The second acquisition module is configured to acquire storage information in each of the target data blocks, and fill the empty data blocks corresponding to each of the target data blocks according to the storage information.

[0040] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0041] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0042] The embodiments of the present application provide a snapshot data processing method, device, electronic device and storage medium, which belong to the field of data storage. The method includes: obtaining multiple snapshots; wherein the multiple snapshots are arranged according to the generation time to form a snapshot chain, each snapshot includes multiple data blocks, and the data blocks include empty data blocks and non-empty data blocks; among the multiple snapshots in the snapshot chain, determining the target flattened snapshot; for each empty data block in the target flattened snapshot, searching in sequence in the snapshot chain along the direction of the generation time earlier than the target flattened snapshot to determine the target data block corresponding to each empty data block; wherein the target data block is a non-empty data block; obtaining the storage information in each target data block, and filling the empty data block corresponding to each target data block according to the storage information. The present application reduces the number of snapshots that need to be traversed when reading data by filling the empty data blocks in the target flattened snapshot. After the empty data blocks in the target flattened snapshot are filled with the data of the target data block, when reading data, the required data can be read directly in the target flattened snapshot without having to continue to search backward in the snapshot chain, thereby reducing the search time and improving the reading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the related technology provided by the embodiments of the present application;

[0044] Figure 2 is a flowchart of a snapshot data processing method provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of snapshot data after flattening provided in an embodiment of the present application;

[0046] Figure 4 is a flowchart of a snapshot data processing method provided by another embodiment of the present application;

[0047] Figure 5 is a flowchart of a snapshot data processing method provided by another embodiment of the present application;

[0048] Figure 6 is a flowchart of a snapshot data processing method provided by another embodiment of the present application;

[0049] Figure 7 is a flowchart of a snapshot data processing method provided by another embodiment of the present application;

[0050] Figure 8 is a flowchart of a snapshot data processing method provided by another embodiment of the present application;

[0051] Figure 9 is a flowchart of a snapshot data processing method provided by another embodiment of the present application;

[0052] Figure 10 is a structural diagram of a snapshot data processing device provided in an embodiment of the present application;

[0053] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, used in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0057] In modern distributed storage systems, snapshot technology is widely used to meet data protection and management requirements. One mainstream implementation is ROW (Redact-On-Write) snapshot technology. ROW snapshots allow users to create instantaneous copies of data without disturbing the original data. This is crucial for scenarios such as data backup, recovery, and multi-version data management.

[0058] In related technologies, ROW snapshot technology still has drawbacks in data read performance. When reading data, if the required data is not in the current snapshot, the system must traverse the snapshot chain until the required data is found. This process can result in significant performance overhead, especially when there are a large number of snapshots and frequent data updates. Furthermore, since each read may require traversing multiple snapshots, read performance is highly volatile.

[0059] The ROW snapshot principle is as follows Figure 1 As shown in the figure, taking a snapshot containing 8 data blocks as an example, the data writing process is first described:

[0060] S1: First, confirm that the snapshot ID of the source volume is 999999.

[0061] S2: After writing data 71, 26, 387, 4456, 71, 79 to data blocks 1, 2, 3, 4, 7, and 8 respectively, a snapshot is taken of the volume, and snapshot 999999 is used as the user snapshot ID.

[0062] S3: The snapshot ID of the source volume is switched to 999998. After writing data 521 to data block 5, a snapshot is taken of the volume, and snapshot 999998 is set as the user snapshot ID.

[0063] S4: The snapshot ID of the source volume is switched to 999997. After writing data 21, 25, 678, and 8 to data blocks 1, 4, 5, and 8, respectively, a snapshot is taken of the volume, using snapshot 999997 as the snapshot ID.

[0064] S5: The snapshot ID of the source volume is switched to 999996. No data is written. A snapshot is taken of the volume, and 999996 is used as the snapshot ID.

[0065] S6: The snapshot ID of the source volume is switched to 999995. After writing data 1232, 27, and 8 to data blocks 1, 2, and 8, respectively, a snapshot is taken of the volume, and snapshot 999995 is used as the snapshot ID.

[0066] S7: The snapshot ID of the source volume is switched to 999994, and data 88 is written to data block 5.

[0067] Next, we describe the source volume data reading process:

[0068] S1: When reading data block 1, it is found that the source volume (snap999994) has not been written. It further reads snap999995 and reads data 1232, ending the process.

[0069] S2: When reading data block 2, it is found that the source volume (snap999994) has not been written. It further reads snap999995 and reads data 27, ending the process.

[0070] S3: When reading data block 3, it is found that the source volume (snap999994) has not been written. It further reads snap999995, snap999996, snap999997, and snap999998 until data 387 is read at snap999999. The process ends.

[0071] S4: When reading data block 4, it is found that the source volume (snap999994) has not been written, so snap999995 and snap999996 are read further until data 25 is read at snap999997, and the process ends.

[0072] S5: When reading data block 5, data 88 is read from the source volume (snap999994), and the process ends.

[0073] S6: When reading data block 6, it is found that the source volume (snap999994) has not been written. Further reading snap999995, snap999996, snap999997, snap999998, and snap999999, no data is read, and the process ends.

[0074] S7: When reading data block 7, it is found that the source volume (snap999994) has not been written, so snap999995, snap999996, snap999997, snap999998 are further read until data 71 is read at snap999999, and the process ends.

[0075] S8: When reading data block 8, it is found that the source volume (snap999994) has not been written. The process further reads snap999995 and reads data 89, ending the process.

[0076] In the current ROW snapshot technology solution, the source volume and all snapshots form a logical chain structure. As shown in the figure above, the chain head is the source volume (snap999994), and the chain tail is the oldest snapshot (snap999999). Unless the operation directly reads snapshot 999999, a logical backward traversal may occur. For example, to read data from source volume BLK3, it is necessary to traverse snapshots 9999994, 999995, 999996, 999997, and 999998 until data 387 is read at snapshot 999999. This read operation requires traversing all snapshots in the chain, affecting overall read performance.

[0077] Based on this, embodiments of the present application provide a snapshot data processing method, device, electronic device, and storage medium, aiming to improve the reading efficiency of snapshots.

[0078] The snapshot data processing method, apparatus, and related equipment provided in the embodiments of the present application are specifically described through the following embodiments. First, the snapshot data processing method in the embodiments of the present application is described.

[0079] The snapshot data processing method provided in the embodiment of the present application relates to the field of data storage. The snapshot data processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the snapshot data processing method, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] Figure 1 This is an optional flowchart of the snapshot data processing method provided in the embodiment of the present application. Figure 2 The method may include but is not limited to steps S201 to S204.

[0083] Step S201: Acquire multiple snapshots.

[0084] Step S202: Determine a target flattened snapshot among multiple snapshots in the snapshot chain.

[0085] Step S203 : For each empty data block in the target leveled snapshot, sequentially search the snapshot chain along the direction of generation time earlier than the target leveled snapshot to determine the target data block corresponding to each empty data block.

[0086] Step S204: acquiring storage information in each target data block, and filling the empty data block corresponding to each target data block according to the storage information.

[0087] Steps S201 to S204, as shown in this embodiment of the present application, reduce the number of snapshots that need to be traversed when reading data by filling empty data blocks in the target flattened snapshot. After the empty data blocks in the target flattened snapshot are filled with data from the target data blocks, the required data can be directly read from the target flattened snapshot when reading data, without having to search backward in the snapshot chain, thereby reducing search time and improving reading efficiency.

[0088] In step S201 of some embodiments, multiple snapshots may be generated over time in a distributed storage system. These snapshots are arranged in the order of their generation, forming a snapshot chain. A snapshot chain is a logical structure that connects snapshots in chronological order, with the most recent snapshot at the end of the chain and the oldest snapshot at the beginning. This structure enables tracking the evolution of data over time and tracing back to any historical state when needed.

[0089] Each snapshot consists of a set of data blocks, which are the basic units used to store data in the storage system. Data blocks can be further divided into two types: empty data blocks and non-empty data blocks.

[0090] Empty data blocks: These are blocks that do not store any user data in the snapshot. This may be because the data block has never been written to, or the data has been deleted. In the ROW snapshot implementation, empty data blocks generally do not occupy additional storage space because they do not contain data that needs to be retained.

[0091] Non-empty data blocks: In contrast to empty data blocks, non-empty data blocks contain user data. During the snapshot process, these data blocks are copied or referenced to ensure that the snapshot accurately reflects the state of the storage volume at a specific point in time.

[0092] In step S202 of some embodiments, the target flattened snapshots are the objects of the optimization operation, and their selection is based on multiple factors, including but not limited to data access patterns, storage space utilization, and the length of the snapshot chain.

[0093] In some embodiments, the following criteria may be considered when selecting a target flattened snapshot:

[0094] Data access frequency: Select frequently accessed snapshots as target snapshots, because optimizing these snapshots can significantly improve data read performance.

[0095] Storage space utilization: Select snapshots with low storage space utilization and flatten them to reduce storage space waste.

[0096] Snapshot chain length: If the snapshot chain is long, selecting snapshots with appropriate sequence numbers for leveling or leveling in segments can reduce the number of snapshots that need to be traversed when reading subsequent data.

[0097] In step S203 of some embodiments, for each empty data block in the target flattened snapshot, the system can record its identification information in the snapshot, such as the logical block address, and search each snapshot one by one along the snapshot chain, starting from the target flattened snapshot. In each snapshot, the data block corresponding to the empty data block is checked to see if it contains valid data. If a corresponding data block containing valid data is found in an earlier snapshot, it is marked as the target data block. This means that the data block contains data that needs to be copied to the empty data block in the target flattened snapshot. Once the target data block is identified, the system records its location and related data information to facilitate data filling in subsequent steps.

[0098] In some embodiments, if no corresponding data block containing valid data is found in all earlier snapshots, that is, all retrieved data blocks are empty data blocks, the empty data blocks in the target leveled snapshot remain unchanged. In this way, when reading the snapshot, if an empty data block is read in the target leveled snapshot, it means that all subsequent data blocks at corresponding positions are empty data blocks, and there is no need to search backward step by step, which reduces the number of queries and saves system computing resources.

[0099] Through the search and identification in step S203, a non-empty data block is found for each empty data block as the target data block. These target data blocks contain the data required by the empty data block and are key resources for snapshot leveling. By copying the data from the target data block to the empty data block, the reliance on the snapshot chain during subsequent data reads is reduced, thereby improving data read efficiency and performance.

[0100] In step S204 of some embodiments, each empty data block in the target flattened snapshot is processed one by one. Based on the correspondence determined in step S203, a non-empty target data block for each empty data block is found from the snapshot chain. These non-empty target data blocks contain the data required by the empty data block. The data in these non-empty data blocks is then copied to the corresponding empty data blocks in the target flattened snapshot, completing the data filling.

[0101] Data population involves more than just simple data copying; it also involves updating metadata. After data copying is complete, the relevant metadata must be updated to ensure that the state of the data block is consistent with the actual stored data. This includes updating the reference count and other relevant properties of the data block to reflect the latest state of the data block.

[0102] For example, Figure 3As shown, the specific flattening operation is as follows;

[0103] S1: Determine that the target snapshot to be flattened is Snap999996. The data blocks of Snap999996 are all empty data blocks. When flattening data block 1, it is found that Snap999996 has not been written. Snap999997 is further searched, and the data is retrieved. The corresponding empty data block 1 in Snap999996 chunkmeta is updated.

[0104] S2: When flattening data block 2, it is found that Snap999996 has not been written. Snap999997 and Snap999998 are further searched, and data is retrieved on Snap999999. The corresponding empty data block 2 in Snap999996 chunkmeta is updated.

[0105] S3: When flattening data block 3, it is found that Snap999996 has not been written. Snap999997 and Snap999998 are further searched, and data is retrieved on Snap999999. The corresponding empty data block 3 in Snap999996 chunkmeta is updated.

[0106] S4: When flattening data block 4, it is found that Snap999996 has not been written. Snap999997 is further searched, the data is retrieved, and the corresponding empty data block 4 in Snap999996 chunkmeta is updated.

[0107] S5: When flattening data block 5, it is found that Snap999996 has not been written. Snap999997 is further searched, the data is retrieved, and the corresponding empty data block 5 in Snap999996 chunkmeta is updated.

[0108] S6: When flattening data block 6, it is found that Snap999996 has not been written. Snap999997, Snap999998, and Snap999999 are further searched. No data is retrieved, so the empty data block 6 remains unchanged.

[0109] S7: When flattening data block 7, it is found that Snap999996 has not been written. Snap999997 and Snap999998 are further searched. Data is retrieved on Snap999999 and the corresponding empty data block 7 in Snap999996 is updated.

[0110] S8: When flattening data block 8, it is found that Snap999996 has not been written. Snap999997 is further searched, the data is retrieved, and the corresponding empty data block 8 in Snap999996 is updated.

[0111] S9: After all BLKs are leveled, mark the leveling completed.

[0112] Data reading process after flattening:

[0113] When reading data block 1, it is found that the source volume (snap999994) has not been written. The process further reads snap999995 and reads data 1232, ending the process.

[0114] When reading data block 2, it is found that the source volume (snap999994) has not been written. The process further reads snap999995 and reads data 27, ending the process.

[0115] When reading data block 3, it is found that the source volume (snap999994) has not been written. Snap999995 and snap999996 are read further. Data 387 is read in snap999996, and the process ends.

[0116] When reading data block 4, it is found that the source volume (snap999994) has not been written. Snap999995 and snap999996 are read further. Data 25 is read in snap999996, and the process ends.

[0117] When reading data block 5, data 88 is read from the source volume (snap999994), and the process ends.

[0118] When reading data block 6, it was found that the source volume (snap999994) had not been written. Snap999995 and snap999996 were further read. No data was read in snap999996, but snap999996 was found to be a flattened snapshot. The process ended.

[0119] When reading data block 7, it is found that the source volume (snap999994) has not been written. Snap999995 and snap999996 are read further. Data 71 is read in snap999996, and the process ends.

[0120] When reading data block 8, it was found that the source volume (snap999994) had not been written. The process further read snap999995 and read data 89, ending the process.

[0121] Through steps S201 to S204, this embodiment of the present application reduces the number of snapshots that need to be traversed when reading data by filling the empty data blocks in the target flattened snapshot. After the empty data blocks in the target flattened snapshot are filled with the data of the target data blocks, the required data can be read directly from the target flattened snapshot when reading data, without having to continue traversing backward in the snapshot chain, thereby reducing search time and improving reading efficiency.

[0122] See also Figure 4 In some embodiments, step S203 may include but is not limited to steps S401 to S403:

[0123] Step S401 : determining the data block sequence number of each empty data block in the target flattened snapshot.

[0124] Step S402 : According to the data block sequence number and the snapshot chain, the data blocks having the same data block sequence number as the empty data block in each snapshot are retrieved in order of generation time from earliest to latest.

[0125] Step S403 : In response to the first non-empty data block being retrieved, the non-empty data block is determined as a target data block corresponding to the empty data block.

[0126] In some embodiments, in step S401, all empty data blocks within the target flattened snapshot are identified and each is assigned a unique data block sequence number. A data block sequence number is an identifier that uniquely locates a data block within the snapshot chain, regardless of whether its contents are empty. This sequence number assignment is the basis for subsequent retrieval operations, ensuring that each specific data block can be accurately located and processed.

[0127] In step S402 of some embodiments, the system searches for data blocks with the same data block sequence number in each snapshot, starting with the oldest snapshot, based on the data block sequence number of each empty data block. This process is reversed, proceeding from the oldest snapshot to the newest, to ensure that the earliest non-empty data version is found. The system checks each snapshot in chronological order until it finds the first corresponding data block containing valid data.

[0128] In step S403 of some embodiments, the first non-empty data block found in a snapshot in the snapshot chain is determined as the target data block for the corresponding empty data block. This determination is based on the data block sequence number retrieved in the previous step and the snapshot generation time sequence. The first non-empty data block found contains the data required to fill the empty data block in the target flattened snapshot.

[0129] Steps S401 to S403 not only improve data retrieval efficiency but also reduce unnecessary traversals of the snapshot chain, optimizing data reading performance and providing a more solid foundation for subsequent data operations. This precise handling of empty data blocks is one of the key technical means for achieving the objectives of this invention, demonstrating its innovative and practical application in improving the performance of distributed storage systems.

[0130] See also Figure 5In some embodiments, step S204 may include but is not limited to steps S501 to S502:

[0131] Step S501: Obtain storage information in each target data block.

[0132] Step S502: copy the data storage address to the storage information of the empty data block corresponding to the target data block.

[0133] In steps S501 to S502 of some embodiments, the storage information in each target data block is first obtained. The storage information generally includes the actual storage location of the data, the size of the data, the version information of the data, and other possible metadata. By accessing the corresponding snapshot in the snapshot chain, the target data block is located and its storage information is read. Next, the data storage address is copied to the storage information of the empty data block corresponding to the target data block. This means updating the metadata of the empty data block in the target flattened snapshot so that it points to the actual storage location of the target data block. In this way, when a data block in the target flattened snapshot needs to be read, the data can be read directly from the updated storage address without further traversing the snapshot chain. This not only improves the efficiency of data reading, but also reduces the consumption of system resources.

[0134] For example, the target flattened snapshot contains an empty data block with sequence number 2. The corresponding target data block determined in step S403 is located in snapshot sequence number 20 in the snapshot chain, with storage address 0x1000. In step S501, the system reads the storage information for data block sequence number 2 in snapshot sequence number 20, including its storage address 0x1000. Then, in step S502, the system copies this storage address 0x1000 to the storage information for data block sequence number 2 in the target flattened snapshot, updating its metadata.

[0135] Through steps S501 and S502, data space is shared among multiple snapshots, and snapshot flattening only involves reading metadata, significantly reducing system overhead. Multiple snapshots can share the same data space. This means that when multiple snapshots contain data blocks with the same content, they can point to the same physical storage location, rather than each snapshot storing a copy of the data. This sharing mechanism significantly reduces storage resource requirements, avoids data redundancy, and improves storage space utilization. For example, if three snapshots all contain the same version of a data block, they can share the storage space for that data block, rather than each storing its own copy. Furthermore, snapshot flattening primarily involves reading and updating metadata, not the data itself. This means that during the flattening operation, the system does not need to read and copy large amounts of data content; it only needs to read and update the data block's storage address and associated metadata. This approach significantly reduces system resource consumption during the flattening operation, including CPU cycles, I / O operations, and memory usage. Since metadata is typically much smaller than the data itself, this approach makes snapshot flattening more efficient and minimizes the impact on system performance.

[0136] In summary, the implementation of steps S501 and S502 not only improves storage space utilization efficiency and reduces data storage redundancy, but also reduces system resource consumption during snapshot flattening operations by processing only metadata. These effects collectively enhance the performance and reliability of the distributed storage system, providing users with a more efficient and cost-effective data management solution.

[0137] See also Figure 6 In some embodiments, the stored information further includes data length and data check code. Before step S502, the following steps may also be performed, including but not limited to steps S601 to S603:

[0138] Step S601: Read the corresponding data content according to the data storage address.

[0139] Step S602: Verify the read data content using the data length and the data verification code.

[0140] Step S603: copy the data length and the data check code to the storage information of the empty data block.

[0141] In step S601 of some embodiments, the data storage address points to the exact location of the data on the physical storage medium. This address is used to access the storage medium and retrieve the actual data stored in the target data block. This step is the basis for subsequent verification work and ensures that the system can obtain the original data for verification.

[0142] In step S602 of some embodiments, the data length is a value indicating the size of the data block, which helps the system confirm the integrity of the read data. A data checksum is a mechanism used to detect errors during data storage or transmission. Common checksums include cyclic redundancy check (CRC) and checksums. The system uses this information to verify the data content to ensure that it has not been tampered with or damaged.

[0143] In step S603 of some embodiments, the data length and data check code are copied to the storage information of the empty data block, ensuring that the storage information of the empty data block not only contains the data storage address, but also the data length and check code, so that in the subsequent data reading and processing process, the system can verify the integrity and accuracy of the data.

[0144] For example, assume the target data block is stored at address 0x1000, and the data length of the data block is 1024 bytes, and the data check code is ABC123. In step S601, the system first accesses address 0x1000 and reads 1024 bytes of data. In step S602, the system verifies that the length of the read data is 1024 bytes and that the check code matches ABC123 to ensure the integrity and accuracy of the data. In step S603, the system copies the 1024-byte data length and check code ABC123 to the storage information of the corresponding empty data block in the target flattened snapshot, completing the storage information update.

[0145] Through steps S601 to S603, the present invention ensures that during the snapshot flattening process, not only is the data storage address updated, but also that the data length and checksum, which are directly related to data integrity and accuracy, are properly processed. This enhances the system's data management capabilities, improves data reliability, and ensures that data errors can be promptly detected and addressed during data reading and processing, thereby maintaining data integrity and system stability.

[0146] See also Figure 7 In some embodiments, before step S203, the following steps may also be performed, but are not limited to, steps S701 to S702:

[0147] Step S701: Obtain a preset snapshot leveling interval and a snapshot sequence number of a target leveling snapshot.

[0148] Step S702 : determining the snapshot span between the target leveled snapshot and the snapshot sequence number and the snapshot leveling interval in the snapshot chain, and determining the snapshots within the snapshot span as the target search range.

[0149] Step S703 : within the target search range, sequentially search the snapshot chain along the direction of snapshots whose generation time is earlier than the target flattened snapshot to determine the target data block corresponding to each empty data block.

[0150] In step S701 of some embodiments, the snapshot leveling interval is a predefined parameter, or a parameter entered by the user in real time, that specifies the frequency of performing the leveling operation within the snapshot chain. For example, if the leveling interval is set to 3, the system will perform a leveling operation every three snapshots. The snapshot sequence number of the target leveled snapshot is an identifier used to uniquely identify a snapshot within the snapshot chain. Obtaining this information is a prerequisite for determining the target search scope.

[0151] In step S702 of some embodiments, the snapshot span between the target leveled snapshot and the snapshot sequence number and the snapshot leveling interval is determined. This span defines the range of snapshots to be considered in the snapshot chain to find the target data block corresponding to the empty data block. The system determines all snapshots within this span as the target search range, which serves as the boundary for subsequent search operations.

[0152] In step S703 of some embodiments, within the target search range, a sequential search is performed along the snapshot chain, along a direction that precedes the target leveled snapshot. This process begins with the target leveled snapshot and proceeds toward the beginning of the snapshot chain, examining each snapshot individually until the predetermined snapshot span is reached. Within each snapshot, the system examines data blocks corresponding to empty data blocks to determine whether they contain valid data. Once the first data block containing valid data is found, the system identifies it as the target data block and uses it for subsequent leveling operations.

[0153] For example, assume that a snapshot chain consists of snapshots numbered 100, 101, 102, 103, and 104, the preset snapshot leveling interval is 2, and the target leveling snapshot is numbered 102. The snapshot span between snapshot 102 and the target snapshot in the snapshot chain is determined, namely, snapshots 101 and 103. These two snapshots are designated as the target search range. Snapshots 101 and 103 are searched sequentially to find the target data block corresponding to the empty data block.

[0154] Through steps S701 to S703, the present invention accurately determines the target search range in the snapshot chain and effectively finds the target data block corresponding to the empty data block within this range. This method not only improves the efficiency of the snapshot flattening operation but also ensures data integrity and consistency, thereby optimizing the performance and reliability of the storage system.

[0155] See also Figure 8 In some embodiments, before step S202, the following steps may also be performed, but are not limited to, steps S801 to S802:

[0156] Step S801: Obtain a preset flattening operation time period and obtain the current time in real time.

[0157] Step S802 : When the current time is in the leveling operation period, the process continues with: determining a target leveling snapshot from among the multiple snapshots in the snapshot chain.

[0158] In step S801 of some embodiments, the leveling operation time period is pre-set by an administrator or user based on system performance requirements and workload. It defines the time period during which the snapshot leveling operation can be performed. This time period can be a specific hour of the day or a specific day of the week. Real-time acquisition of the current time is key to determining whether to execute the leveling operation, ensuring that the operation is performed only at the appropriate time.

[0159] In some embodiments, in step S802, the current time is compared with a preset leveling time period. If the current time is within the leveling time period, steps S202 to S204 are executed to perform the snapshot leveling operation, including determining the target leveling snapshot, retrieving the target data block, and updating the storage information of the empty data block. If the current time is not within the leveling time period, the system suspends the leveling operation until the next appropriate time period.

[0160] For example, the preset leveling time period is from 10 PM to 2 AM daily to avoid impacting system performance during daytime working hours. In step S801, the system retrieves this time period and obtains the current time in real time. Assuming the current time is 11 PM, it falls within the preset leveling time period. In step S802, the system confirms that the current time is suitable for leveling, and therefore proceeds to steps S202 through S204 to perform the snapshot leveling operation.

[0161] By executing steps S801 and S802, the present invention intelligently performs snapshot leveling within a preset time period, thereby optimizing the data storage structure and improving data access efficiency while ensuring system performance. This approach not only improves the overall performance of the storage system but also enhances system stability and reliability, providing users with a more efficient and flexible data management solution.

[0162] See also Figure 9 In some embodiments, step S202 may also include but is not limited to steps S901 to S904:

[0163] Step S901: Obtain a preset non-empty data block ratio threshold.

[0164] Step S902: For each snapshot in the snapshot chain, obtain the number of non-empty data blocks in the snapshot and the preset total number of data blocks.

[0165] Step S903 : Calculate the ratio between the number of non-empty data blocks and the total number of data blocks to obtain the proportion of non-empty data blocks in the snapshot.

[0166] Step S904 : When the non-empty data block ratio of the snapshot is less than or equal to the non-empty data block ratio threshold, the snapshot is determined as a target flattened snapshot.

[0167] In step S901 of some embodiments, the non-empty data block ratio threshold is a predefined parameter used to evaluate whether the ratio of non-empty data blocks in the snapshot meets the leveling requirement. The threshold can be set based on various factors, such as storage space utilization, data access patterns, or system performance requirements.

[0168] In step S902 of some embodiments, the total number of data blocks refers to the total number of blocks that can store data in the snapshot, and the number of non-empty data blocks refers to the number of blocks that actually contain data. This information is the basis for calculating the non-empty data block ratio.

[0169] In some embodiments, in step S903, the ratio of the number of non-empty data blocks in each snapshot to the total number of data blocks is calculated to obtain the non-empty data block ratio of the snapshot. This ratio reflects the utilization efficiency of the data blocks in the snapshot and is an important indicator for evaluating whether a flattening operation is necessary.

[0170] In step S904 of some embodiments, the calculated non-empty data block ratio is compared with a preset threshold. If the non-empty data block ratio of a snapshot is less than or equal to the non-empty data block ratio threshold, this indicates that the snapshot contains a large number of empty data blocks and that a leveling operation can improve data block usage efficiency. Therefore, the system determines the snapshot as a target leveled snapshot.

[0171] For example, the preset non-empty data block ratio threshold is 20%. That is, if the ratio of non-empty data blocks in a snapshot does not exceed 20%, the snapshot is considered suitable for leveling. In step S902, the system discovers that snapshot A has two non-empty data blocks, while the preset total number of data blocks is 10. In step S903, the system calculates that the non-empty data block ratio of snapshot A is 20%. Since this ratio is equal to the preset threshold, the system determines snapshot A as the target leveling snapshot in step S904.

[0172] By executing steps S901 to S904, the present invention can intelligently select a target flattened snapshot based on the usage efficiency of the data blocks, thereby optimizing the usage of storage space and improving system performance.

[0173] See also Figure 10The embodiment of the present application further provides a snapshot data processing device that can implement the above-mentioned snapshot data processing method, and the device includes:

[0174] A first acquisition module is configured to acquire multiple snapshots, wherein the multiple snapshots are arranged according to generation time to form a snapshot chain, each snapshot includes multiple data blocks, and the data blocks include empty data blocks and non-empty data blocks;

[0175] A determination module, configured to determine a target flattened snapshot among multiple snapshots in the snapshot chain;

[0176] a search module configured to sequentially search, for each empty data block in the target flattened snapshot, the snapshot chain along a direction of generation time earlier than the target flattened snapshot, to determine a target data block corresponding to each empty data block; wherein the target data block is a non-empty data block;

[0177] The second acquisition module is used to acquire storage information in each target data block and fill the empty data block corresponding to each target data block according to the storage information.

[0178] The specific implementation of the image compliance detection device is basically the same as the specific embodiment of the above-mentioned snapshot data processing method, and will not be repeated here.

[0179] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned snapshot data processing method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0180] See also Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0181] The processor 1101 may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0182] The memory 1102 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called by the processor 1101 to execute the snapshot data processing method of the embodiments of this application.

[0183] Input / output interface 1103, used to implement information input and output;

[0184] Communication interface 1104, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0185] Bus 1105 , which transmits information between various components of the device (e.g., processor 1101 , memory 1102 , input / output interface 1103 , and communication interface 1104 );

[0186] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via a bus 1105 .

[0187] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned snapshot data processing method is implemented.

[0188] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The embodiments of the present application provide a snapshot data processing method, device, electronic device and storage medium, which belong to the field of data storage. The method includes: obtaining multiple snapshots; wherein the multiple snapshots are arranged according to the generation time to form a snapshot chain, each snapshot includes multiple data blocks, and the data blocks include empty data blocks and non-empty data blocks; among the multiple snapshots in the snapshot chain, determining the target flattened snapshot; for each empty data block in the target flattened snapshot, searching in sequence in the snapshot chain along the direction of the generation time earlier than the target flattened snapshot to determine the target data block corresponding to each empty data block; wherein the target data block is a non-empty data block; obtaining the storage information in each target data block, and filling the empty data block corresponding to each target data block according to the storage information. The present application reduces the number of snapshots that need to be traversed when reading data by filling the empty data blocks in the target flattened snapshot. After the empty data blocks in the target flattened snapshot are filled with the data of the target data block, when reading data, the required data can be read directly in the target flattened snapshot without having to continue to search backward in the snapshot chain, thereby reducing the search time and improving the reading efficiency.

[0190] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0191] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0193] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0194] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0195] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0200] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A snapshot data processing method, characterized in that: include: Acquire multiple snapshots; wherein the multiple snapshots are arranged according to the generation time to form a snapshot chain, each of the snapshots includes multiple data blocks, and the data blocks include empty data blocks and non-empty data blocks; Determining a target flattened snapshot among the plurality of snapshots in the snapshot chain; Obtaining a preset snapshot leveling interval and a snapshot sequence number of the target leveled snapshot, determining a snapshot span between the target leveled snapshot and the snapshot chain according to the snapshot sequence number and the snapshot leveling interval, and determining the snapshots within the snapshot span as a target search range; Determining a data block sequence number of each of the empty data blocks in the target flattened snapshot; According to the data block sequence number and the snapshot chain, searching for the data block with the same data block sequence number as the empty data block in each snapshot within the target search range in descending order of generation time; In response to retrieving a first non-empty data block, determining the non-empty data block as a target data block corresponding to the empty data block; The storage information in each of the target data blocks is acquired, and the empty data blocks corresponding to each of the target data blocks are filled according to the storage information.

2. The method according to claim 1, characterized in that The acquiring storage information in each of the target data blocks and filling the empty data blocks corresponding to each of the target data blocks according to the storage information includes: Acquire the storage information in each target data block; wherein the storage information includes a data storage address; The data storage address is copied to the storage information of the empty data block corresponding to the target data block.

3. The method according to claim 2, characterized in that The storage information further includes a data length and a data check code, and before copying the data storage address to the empty data block corresponding to the target data block, further includes: Read the corresponding data content according to the data storage address; Verify the read data content using the data length and the data check code; The data length and the data check code are copied to the storage information of the empty data block.

4. The method according to claim 1, wherein Before determining a target leveled snapshot among the plurality of snapshots in the snapshot chain, the method further includes: Get the preset flattening operation time period and get the current time in real time; When the current time is in the flattening operation time period, continue to execute: A target flattened snapshot is determined among the plurality of snapshots in the snapshot chain.

5. The method according to claim 1, characterized in that Determining a target leveled snapshot among the plurality of snapshots in the snapshot chain includes: Get the preset non-empty data block ratio threshold; For each snapshot in the snapshot chain, obtain the number of non-empty data blocks in the snapshot and a preset total number of data blocks; Calculating the ratio of the number of non-empty data blocks to the total number of data blocks to obtain a non-empty data block ratio of the snapshot; When the non-empty data block ratio of the snapshot is less than or equal to the non-empty data block ratio threshold, the snapshot is determined as the target flattened snapshot.

6. A snapshot data processing device, characterized in that: include: A first acquisition module is configured to acquire a plurality of snapshots, wherein the plurality of snapshots are arranged according to generation time to form a snapshot chain, each of the snapshots comprising a plurality of data blocks, wherein the data blocks comprise empty data blocks and non-empty data blocks; a determination module, configured to determine a target leveled snapshot among the plurality of snapshots in the snapshot chain; a retrieval module configured to obtain a preset snapshot leveling interval and a snapshot sequence number of the target leveled snapshot, determine a snapshot span between the target leveled snapshot and the snapshot chain based on the snapshot sequence number and the snapshot leveling interval, and determine the snapshots within the snapshot span as a target retrieval range; determine a data block sequence number of each empty data block in the target leveled snapshot; sequentially search for data blocks with the same data block sequence number as the empty data block in each snapshot within the target retrieval range based on the data block sequence number and the snapshot chain in descending order of generation time; and in response to retrieving a first non-empty data block, determine the non-empty data block as a target data block corresponding to the empty data block; The second acquisition module is configured to acquire storage information in each of the target data blocks, and fill the empty data blocks corresponding to each of the target data blocks according to the storage information.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the snapshot data processing method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the snapshot data processing method according to any one of claims 1 to 5 is implemented.

Images