Data processing method, device and system

Abstract

The present invention discloses a method, device and system for data processing. The method comprises: receiving a task for calculating occupied space; obtaining the number of occupied data blocks of a specified snapshot in the task for calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot; obtaining the total number of occupied data blocks in the task for calculating occupied space according to the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots. The present invention solves the technical problem that the existing technology consumes a large amount of computing resources in calculating the occupied space of snapshots, thereby affecting the normal operation of cloud service business.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, device and system for data processing. Background Art

[0002] In the existing cloud service business, cloud disk snapshots need to occupy OSS or file system storage resources. Therefore, cloud disk snapshots are charged based on the total space occupied by the snapshots in GB. The existing algorithm for calculating the space occupied by snapshots is to read the index map of all snapshots of the cloud disk and insert each data block information in the index map into the data structure collection. The uniqueness of the key value of the collection is used to deduplicate the snapshot data blocks in memory to obtain the logical space actually occupied by the snapshot.

[0003] It can be seen from this that the current method for calculating the space occupied by snapshots requires a large amount of additional space and bandwidth resources. When calculating the space occupied by multiple cloud disk snapshots at the same time, a large amount of system resources will be consumed, affecting normal business.

[0004] With regard to the above-mentioned problem that the existing technology consumes a large amount of computing resources in calculating the space occupied by the snapshot, thereby affecting the normal operation of the cloud service business, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and system for data processing, so as to at least solve the technical problem that the conventional technology consumes a large amount of computing resources in computing the space occupied by snapshots, thereby affecting the normal operation of cloud service business.

[0006] According to one aspect of an embodiment of the present invention, a data processing system is provided, comprising: a scheduling device and a data processing device, wherein the scheduling device is used to receive a snapshot occupied space request; generate at least one occupied space calculation task based on the snapshot occupied space request, and send each occupied space calculation task to the corresponding data processing device; the data processing device is used to receive the occupied space calculation task, obtain the number of occupied data blocks of a specified snapshot in the occupied space calculation task, and the number of difference data blocks between the specified snapshot and the parent snapshot of the specified snapshot; obtain the total number of occupied data blocks in the occupied space calculation task based on the number of occupied data blocks and the number of difference data blocks, wherein the specified snapshot and the parent snapshot are adjacent snapshots.

[0007] According to another aspect of an embodiment of the present invention, a method for data processing is also provided, including: receiving a task for calculating occupied space; obtaining the number of occupied data blocks of a specified snapshot in the task for calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot; obtaining the total number of occupied data blocks in the task for calculating occupied space based on the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots.

[0008] Optionally, obtaining the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot includes: obtaining the number of occupied data blocks of the specified snapshot from a preset index mapping table; determining whether the first parent snapshot of the specified snapshot exists; if the judgment result is yes, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot based on the status of the first parent snapshot in the preset index mapping table; if the judgment result is no, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks.

[0009] Further, optionally, based on the status of the first parent snapshot in the preset index mapping table, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot includes: if the first parent snapshot is not in the preset index mapping table, traversing the preset snapshot chain to obtain the second parent snapshot; obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot; if the first parent snapshot is in the preset index mapping table, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot.

[0010] Optionally, obtaining the second parent snapshot includes: obtaining a snapshot whose distance from the specified snapshot is less than a preset distance; and determining the snapshot as the second parent snapshot.

[0011] Optionally, obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot includes: obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot based on whether the snapshot identifiers managed by the data blocks in a preset index mapping table are the same; or obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot based on fingerprint information of the data blocks.

[0012] Further, optionally, the method further comprises: updating the difference data information in the preset index mapping table, wherein the difference data information is used to indicate the number of data blocks of the snapshot that is different from the specified snapshot. Optionally, the method further comprises: generating occupied space response information according to the total number of occupied data blocks; and sending the occupied space response information to the scheduling device.

[0013] Further, optionally, generating occupied space response information based on the total number of occupied data blocks includes: obtaining the number of occupied data blocks from each data block management respectively; calculating the space occupied by the number of occupied data blocks of each data block management to obtain the total number of occupied data blocks; generating occupied space response information based on the total number of occupied data blocks.

[0014] According to another aspect of an embodiment of the present invention, another data processing method is provided, including: receiving a snapshot occupied space request; generating at least one calculated occupied space task based on the snapshot occupied space request; and sending each calculated occupied space task to a corresponding data processing device.

[0015] Optionally, the method further includes: receiving occupied space response information sent by at least one data processing device; and counting the total number of occupied data blocks based on the occupied space response information.

[0016] According to another aspect of an embodiment of the present invention, a data processing device is also provided, including: a receiving module for receiving a task for calculating occupied space; an acquisition module for acquiring the number of occupied data blocks of a specified snapshot in the task for calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot; a calculation module for obtaining the total number of occupied data blocks in the task for calculating occupied space based on the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots.

[0017] According to another aspect of an embodiment of the present invention, another data processing device is provided, including: a receiving module for receiving a snapshot occupied space request; a task generating module for generating at least one calculation occupied space task based on the snapshot occupied space request; and a sending module for sending each calculation occupied space task to a corresponding data processing device.

[0018] According to another aspect of the embodiments of the present invention, a storage medium is provided. The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned data processing method.

[0019] According to yet another aspect of the embodiments of the present invention, a processor is provided. The processor is used to run a program, wherein the above-mentioned data processing method is executed when the program is run.

[0020] In an embodiment of the present invention, a task of calculating occupied space is received; the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot are obtained; the total number of occupied data blocks in the task of calculating occupied space is obtained according to the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots, thereby achieving the purpose of reducing the impact of the algorithm for calculating the occupied space of the snapshot on the system resources, thereby achieving the technical effect of improving the calculation speed of the occupied space of the snapshot, and further solving the technical problem that the prior art consumes a large amount of computing resources in calculating the occupied space of the snapshot, thereby affecting the normal operation of the cloud service business. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 is a schematic diagram of a data processing system according to Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of snapshot storage in a data processing system according to Embodiment 1 of the present invention;

[0024] Figure 3 It is a hardware structure block diagram of a computer terminal of a data processing method according to an embodiment of the present invention;

[0025] Figure 4 is a flowchart of a data processing method according to a second embodiment of the present invention;

[0026] Figure 5 is a flowchart of calculating the total number of data blocks in the data processing method according to the second embodiment of the present invention;

[0027] Figure 6 is a flowchart of a data processing method according to Embodiment 3 of the present invention;

[0028] Figure 7 is a schematic diagram of a data processing device according to a fourth embodiment of the present invention;

[0029] Figure 8 is a schematic diagram of a data processing device according to a fifth embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Technical terms involved in this application:

[0033] Virtual machine disk: virtual machine block device, referred to as cloud disk;

[0034] OSS: object storage service;

[0035] Block Master: responsible for task scheduling and request processing service roles;

[0036] Block Server: a service role responsible for reading, writing and processing data on cloud disk block devices;

[0037] Segment: a unit where the cloud disk is logically divided into 32G units;

[0038] Cloud disk snapshot: a record of data changes on a cloud disk at a certain point in time;

[0039] Standard snapshot: cloud disk block device snapshot, and upload to OSS;

[0040] Second-level snapshot: A snapshot of a cloud disk block device. The data is stored in the file system and is available in seconds.

[0041] Snapshot data block (Block): The logical space of the cloud disk is divided into 2MB bytes;

[0042] Snapshot space: the logical space occupied by all existing snapshots in the file system;

[0043] Snapshot effective space: the sum of non-empty 2MB data blocks in the snapshot;

[0044] Snapshot incremental data: data changes between the last snapshot and the current snapshot;

[0045] Snapshot data differences: data blocks that are not shared between two snapshots;

[0046] Index Map: a data structure that records the correspondence between snapshot data blocks and the actual snapshots they are in;

[0047] BlockIdType: data structure that records snapshot block information in OSS snapshots;

[0048] SnapshotId: snapshot identification information, uniquely identifies a snapshot;

[0049] BlockId: the logical index value of the data block in the snapshot, starting from zero;

[0050] Snapshot Metadata: Metadata information of the snapshot, including the identifier of each snapshot data block and the snapshot information to which the snapshot data block belongs.

[0051] Example 1

[0052] According to one aspect of an embodiment of the present invention, a data processing system is provided. Figure 1 is a schematic diagram of a data processing system according to Embodiment 1 of the present invention. Figure 1 As shown, it includes: a scheduling device 12 and a data processing device 14, wherein the scheduling device 12 is used to receive a snapshot occupied space request; generate at least one computing occupied space task according to the snapshot occupied space request, and send each computing occupied space task to the corresponding data processing device; the data processing device 14 is used to receive the computing occupied space task, obtain the number of occupied data blocks of a specified snapshot in the computing occupied space task, and the number of difference data blocks between the specified snapshot and the parent snapshot of the specified snapshot; obtain the total number of occupied data blocks in the computing occupied space task according to the number of occupied data blocks and the number of difference data blocks, wherein the specified snapshot and the parent snapshot are adjacent snapshots.

[0053] Specifically, the data processing system provided by the embodiment of the present application can be applied to a cloud storage system, wherein the snapshot in the cloud storage system needs to occupy an object storage service, so the scheduling device 12 in the data processing system provided by the embodiment of the present application receives a snapshot space request, generates at least one computing space task based on the snapshot space request, and sends at least one computing space task to the data processing device 14, wherein each computing space task corresponds to a data processing device 14, and since all computing space tasks are calculated in parallel, the time used for the entire calculation is roughly equivalent to the difference in data blocks of a single segment when the processing capacity is sufficient. The entire calculation time can be completed in seconds, which reduces the calculation time relative to the calculation time of the previous algorithm.

[0054] The data processing system provided in the embodiment of the present application calculates the space occupied by the snapshot as follows:

[0055] Step 1, when the Block Master (i.e., the scheduling device 12 in the embodiment of the present application) receives a request to query the occupied space of a snapshot, it calculates the snapshot pair that needs to be updated (the new parent snapshot S1 and the snapshot S2 that needs to be updated) according to the snapshot list input by the management;

[0056] Step 2, Block Master calculates the number of snapshot segments (i.e., the task of calculating occupied space in the embodiment of the present application) and sends it to the corresponding Block server (i.e., the data processing device 14 in the embodiment of the present application) to complete the calculation of the data difference of each segment;

[0057] Step 3, Block Server reads the Index Map information of the Segment in S2, counts the size of the difference data blocks, and updates it to Block Master;

[0058] Step 4: Block Master waits for all segments to be updated, summarizes the occupied space of all segments, and completes the query request.

[0059] The Block Master provided in the embodiment of the present application is responsible for task scheduling and request processing services, and the Block Server is responsible for reading, writing and processing cloud disk block device data. In the embodiment of the present application, only the Block Master is used as a scheduling device, and the Block Server is used as a data processing device for explanation. The data processing system provided in the embodiment of the present application is used as the basis, and no specific limitation is made. The calculation algorithm for calculating the total snapshot space occupied by the data processing system in the embodiment of the present application is the sum of the effective space of the first snapshot and the difference between other snapshots and the parent snapshot.

[0060] The creation and deletion process of snapshots and the changes in snapshot information are as follows: Figure 2 As shown, Figure 2 is a schematic diagram of snapshot storage in a data processing system according to Embodiment 1 of the present invention;

[0061] like Figure 2 As shown, Figure 2 The first set of snapshot storage conditions in the example includes: 6 snapshots, respectively represented as snapshot1, snapshot2, snapshot3, snapshot4, snapshot5 and snapshot6. The eight data blocks marked as 1 in snapshot1 are the space occupied by snapshot1. The storage conditions of the eight data blocks in snapshot2 have changed, such as Figure 2 As shown in the first group of snapshots in , the number of snapshot difference data blocks between snapshot1 and snapshot2 is 2. Therefore, in the snapshot space occupied table (i.e., snapshot list) recorded in the first group of tables, the number of snapshot difference data blocks recorded between snapshot2 and snapshot1 is 2. Here, snapshot1 is the parent snapshot of snapshot2. Similarly, snapshot2 is the parent snapshot of snapshot3, and the number of snapshot difference data blocks is 2. Similarly, the relationship between snapshots and parent snapshots from snapshot1 to snapshot6, as well as the number of snapshot difference data blocks, are obtained, and the incremental relationship of the space occupied by the snapshots is obtained based on the number of snapshot difference data blocks.

[0062] Figure 2 The second group of snapshots records the snapshots at the beginning of the deletion snapshot chain, the third group records the snapshots in the middle of the deletion snapshot chain, and the fourth group records the snapshots at the end of the deletion snapshot chain. When calculating the space occupied by the snapshots, the snapshot chain is traversed according to the snapshot list passed by the control. If the deleted snapshot is a specified snapshot in the snapshot chain, the relative incremental relationship of the snapshots after the deleted snapshot needs to be recalculated. The calculation method is as follows:

[0063] 1. The deleted snapshot is the snapshot at the beginning of the snapshot chain.

[0064] Find the first undeleted snapshot and calculate the total number of occupied data blocks by adding the number of its valid data blocks to the incremental data blocks of all snapshots.

[0065] 2. The deleted snapshot is the snapshot at the end of the snapshot chain.

[0066] The total number of occupied data blocks is calculated by summing the incremental data blocks of all snapshots before the snapshot is deleted.

[0067] 3. The deleted snapshot is the snapshot in the middle of the snapshot chain.

[0068] a) Find the snapshot S1 that is closest to the deleted snapshot as the parent snapshot of the snapshot S2 that is after the deleted snapshot; Figure 2 As shown in the third group of snapshots in , if snapshot3 and snapshot5 are deleted, the parent snapshot of snapshot4 is changed to snapshot2, and the parent snapshot of snapshot6 is changed to snapshot4.

[0069] b) Read the Index Map of S2 and count the number of data blocks that are different from those of S1. The method for calculating the difference can be to compare whether the snapshot IDs of the data blocks in the Index Map are the same, or to compare based on the fingerprint information of the data blocks. For example, Figure 2 As shown in the third group of snapshots in , the number of difference data blocks between snapshot4 and snapshot2 is 5; the number of difference data blocks between snapshot6 and snapshot4 is 4.

[0070] c) Update the data difference information of snapshot S2 and calculate the total occupied space. Based on steps a and b, the total occupied space is calculated, wherein the number of superimposed difference data blocks is 17, and the total snapshot occupied space is updated based on the number of difference data blocks.

[0071] Example 2

[0072] According to an embodiment of the present invention, a method embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 3 FIG. 1 is a hardware structure block diagram of a computer terminal of a data processing method according to an embodiment of the present invention. Figure 3 As shown, the computer terminal 20 may include one or more (only one is shown in the figure) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission module 306 for communication functions. It can be understood by those skilled in the art that Figure 3The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0074] The memory 304 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the data processing method in the embodiment of the present invention. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, the data processing method of the above-mentioned application program is realized. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the computer terminal 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0075] The transmission module 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 30. In one example, the transmission module 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0076] Under the above operating environment, this application provides Figure 4 The data processing method shown. On the data processing device side, Figure 4 is a flow chart of a data processing method according to a second embodiment of the present invention.

[0077] Step S402, receiving a task for calculating occupied space;

[0078] In the above step S402 of the present application, the data processing device receives a task for calculating occupied space sent by the scheduling device, wherein the task for calculating occupied space includes calculating the total occupied space of the snapshot.

[0079] Step S404, obtaining the number of occupied data blocks of the specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot;

[0080] In the above step S404 of the present application, the data processing device obtains the number of occupied data blocks of the specified snapshot in the task of calculating the occupied space according to the task of calculating the occupied space. For example, in Example 1 Figure 2 The groups of snapshots shown are Figure 2 Take the first set of snapshots in as an example, assuming that the specified snapshot is snapshot6. Figure 2 It can be seen that the number of data blocks occupied by snapshot6 is 8, the parent snapshot of snapshot6 is snapshot5, and the number of different data blocks between snapshot6 and snapshot5 is 2.

[0081] Step S406, obtaining the total number of occupied data blocks in the task of calculating occupied space according to the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot is a snapshot adjacent to the first parent snapshot.

[0082] In the above step S406 of the present application, the data processing device traverses the snapshot chain according to the number of occupied data blocks and the number of difference data blocks of the specified snapshot obtained in step S404, and finally obtains the total number of occupied data blocks in the task of calculating the occupied space.

[0083] As in Example 1 Figure 2 As shown in the first group of snapshots in , the snapshot chain includes “snapshot6→snapshot5→snapshot4→snapshot3→snapshot2→snapshot1”. Based on the snapshot chain, the difference data block data stream between each snapshot and the parent snapshot is superimposed to finally obtain the total number of occupied data blocks in the task of calculating the occupied space.

[0084] In the embodiment of the present application, specifying a snapshot as a snapshot adjacent to the first snapshot includes: Figure 2 Taking the first group of snapshots shown as an example, the parent snapshot of snapshot6 is snapshot5, the parent snapshot of snapshot5 is snapshot4, the parent snapshot of snapshot4 is snapshot3, the parent snapshot of snapshot3 is snapshot2, and the parent snapshot of snapshot2 is snapshot1.

[0085] In an embodiment of the present invention, a task of calculating occupied space is received; the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot are obtained; the total number of occupied data blocks in the task of calculating occupied space is obtained according to the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots, thereby achieving the purpose of reducing the impact of the algorithm for calculating the occupied space of the snapshot on the system resources, thereby achieving the technical effect of improving the calculation speed of the occupied space of the snapshot, and further solving the technical problem that the prior art consumes a large amount of computing resources in calculating the occupied space of the snapshot, thereby affecting the normal operation of the cloud service business.

[0086] Optionally, obtaining the number of occupied data blocks of the specified snapshot in the task of calculating occupied space and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot in step S404 includes:

[0087] Step S4041, obtaining the number of occupied data blocks of a specified snapshot from a preset index mapping table;

[0088] In the above step S4041 of the present application, assuming that the specified snapshot is snapshot6, the number of occupied data blocks of the specified snapshot is obtained from the preset index mapping table as shown in Table 1:

[0089] Table 1

[0090] snapshotID ParentSnapshotID totalBlocks changedBlocks snapshot6 snapshot5 8 2 4 bytes 4 bytes 4 bytes 1 bytes

[0091] Among them, the first row in Table 1 is the snapshot ID, the parent snapshot ID, the number of valid data blocks, and the number of snapshot difference data blocks. The corresponding second row is the size of the space occupied by each type of snapshot / data block. It can be seen from Table 1 that the number of data blocks occupied by snapshot6 (that is, the number of valid data blocks in Table 1) is 8 blocks.

[0092] Step S4042, determining whether the first parent snapshot of the specified snapshot exists;

[0093] Step S4043: if the result of the judgment is yes, the number of difference data blocks between the first parent snapshot and the specified snapshot is obtained according to the state of the first parent snapshot in the preset index mapping table;

[0094] In the above step S4043 of the present application, as shown in Table 1, the first parent snapshot of snapshot6 is snapshot5, and the number of difference data blocks between snapshot5 and snapshot6 is 2.

[0095] Step S4044: if the judgment result is no, the number of occupied data blocks of the specified snapshot is determined as the total number of occupied data blocks.

[0096] In the above step S4044 of the present application, snapshot6 determines that the snapshot chain has been updated to only snapshot6 when there is no first parent snapshot, so the number of occupied data blocks of snapshot6 (8 data blocks) is the final total number of occupied data blocks.

[0097] In summary, in combination with step S4041 to step S4044, after obtaining the number of occupied data blocks of the specified snapshot from the preset index mapping table, it is determined whether the first parent snapshot of the specified snapshot exists, if so, step S4043 is executed, if not, step S4044 is executed.

[0098] Further, optionally, in step S4043, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot according to the state of the first parent snapshot in the preset index mapping table includes:

[0099] Step S40431: if the first parent snapshot is not in the preset index mapping table, traverse the preset snapshot chain to obtain the second parent snapshot;

[0100] In the above step S40431 of this application, if Figure 2 As shown in the figure, if the first parent snapshot of snapshot6 is not in the preset index mapping table, then according to Figure 2 The preset snapshot chain in is traversed to obtain the second parent snapshot of snapshot6, that is, the new parent snapshot.

[0101] Step S40432, obtaining the number of differential data blocks between the second parent snapshot and the specified snapshot;

[0102] In the above step S40432 of this application, if Figure 2 As shown in the figure, if snapshot5 is deleted, the parent snapshot closest to snapshot6 is snapshot4. Figure 2 As shown, the number of different data blocks between snapshot4 and snapshot6 is 4 data blocks.

[0103] Step S40433: if the first parent snapshot is in the preset index mapping table, the number of difference data blocks between the first parent snapshot and the specified snapshot is obtained.

[0104] In the above step S40433 of this application, if Figure 2 As shown, if the first parent snapshot of snapshot6 is in the preset index mapping table, then according to Figure 2 Traverse the preset snapshot chain in and get the first parent snapshot snapshot5 of snapshot6. Figure 2 It can be seen that the number of different data blocks between snapshot5 and snapshot6 is 2 data blocks.

[0105] Optionally, obtaining the second parent snapshot in step S40431 includes:

[0106] Step S404311, obtaining a snapshot whose distance from the specified snapshot is less than a preset distance;

[0107] Step S404312: determine the snapshot as the second parent snapshot.

[0108] Specifically, in combination with step S404311 and step S404312, based on step S40431, the second parent snapshot of snapshot6 is obtained by obtaining the snapshot closest to snapshot6 in the preset snapshot chain, and using the snapshot as the new parent snapshot of snapshot6. Figure 2 As shown, after snapshot5 is deleted, the snapshot closest to snapshot6 is snapshot4, so the new parent snapshot of snapshot6 is snapshot4, that is, the second parent snapshot.

[0109] Optionally, obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot in step S40432 includes:

[0110] According to whether the snapshot identifiers of the data block management in the preset index mapping table are the same, the number of difference data blocks between the second parent snapshot and the specified snapshot is obtained;

[0111] or,

[0112] The number of difference data blocks between the second parent snapshot and the specified snapshot is obtained according to the fingerprint information of the data blocks.

[0113] Specifically, in the embodiment of the present application, the process of obtaining the number of differential data blocks between the second parent snapshot and the specified snapshot includes two methods. Method 1 is to obtain the number of differential data blocks between the second parent snapshot and the specified snapshot based on whether the snapshot identifiers managed by the data blocks in the preset index mapping table are the same; Method 2 is to obtain the number of differential data blocks between the second parent snapshot and the specified snapshot based on the fingerprint information of the data blocks.

[0114] In the first method, the data blocks of snapshot 4 are read from the preset index mapping table, and compared with the data blocks of snapshot 6 to obtain that the number of different data blocks between snapshot 4 and snapshot 6 is 4 data blocks.

[0115] Method 2: Compare based on the fingerprint information of the data block, that is, the summary value of the data block, or the hash value.

[0116] Further, optionally, the data processing method provided in the embodiment of the present application also includes:

[0117] Step S4045: updating the difference data information in the preset index mapping table, wherein the difference data information is used to indicate the number of data blocks of the snapshot that is different from the specified snapshot.

[0118] Specifically, Figure 2 As shown, after snapshot5 is deleted, the difference data information in the preset index mapping table is updated to obtain Table 2.

[0119] Table 2

[0120] snapshotID ParentSnapshotID totalBlocks changedBlocks snapshot6 Snapshot 4 8 4 4 bytes 4 bytes 4 bytes 2 bytes

[0121] Specifically, Figure 5 FIG. 4 is a flowchart of calculating the total number of data blocks in the data processing method according to the second embodiment of the present invention. Figure 5 As shown, a task of calculating occupied space is obtained, and it is determined whether the snapshot chain in the task of calculating occupied space (wherein the snapshot chain is included in the preset index mapping table in the embodiment of the present application) is empty. If not, the last snapshot of the snapshot chain is obtained, and it is determined whether the snapshot exists. If so, the parent snapshot of the snapshot is obtained, and it is determined whether the parent snapshot exists. If the parent snapshot exists, the number of occupied data blocks of the snapshot and the amount of differential data blocks between the snapshot and the parent snapshot are calculated; if the parent snapshot does not exist, the snapshot closest to the snapshot is obtained forward according to the snapshot chain, and the snapshot closest to the snapshot is determined as the new parent snapshot (that is, the second parent snapshot in the embodiment of the present application). Similarly, the number of occupied data blocks of the snapshot and the number of differential data blocks between the snapshot and the new parent snapshot are calculated; similarly, the entire snapshot chain is traversed in the above manner to obtain the total occupied quantity.

[0122] like Figure 5 As shown, the process of calculating the total number of data blocks in the data processing method provided in the embodiment of the present application is specifically as follows:

[0123] Step 1, query the usage of the snapshot list; that is, query the usage of the preset index mapping table in the embodiment of the present application.

[0124] Step 2, determine whether the preset index mapping table is empty; if so, end the process; if not, continue querying and execute Step 3;

[0125] Step 3, get the last snapshot, whether the snapshot exists; if it exists, execute Step 4; if it does not exist, remove the snapshot from the snapshot chain of the preset index mapping table and return to Step 2;

[0126] Step 4, obtain the parent snapshot of the last snapshot, and determine whether the last snapshot is in the parent snapshot of the snapshot chain. If the parent snapshot does not exist, the number of occupied data blocks occupied by the last snapshot is calculated to obtain the total number of occupied data blocks, and the process ends; if the parent snapshot exists, execute Step 5;

[0127] Step 5, determine whether the parent snapshot exists in the snapshot chain of the preset index mapping table; if so, calculate the number of occupied data blocks of the last snapshot and the amount of data blocks that differ between the last snapshot and the parent snapshot; if not, execute Step 6;

[0128] Step 6: Obtain the snapshot closest to the snapshot based on the snapshot chain, and determine the snapshot closest to the snapshot as the new parent snapshot. Similarly, calculate the number of occupied data blocks of the snapshot and the number of different data blocks between the snapshot and the new parent snapshot.

[0129] Similarly, if Figure 5 As shown, the entire snapshot chain is traversed according to the above method to obtain the total occupied quantity.

[0130] Optionally, the data processing method designed in the embodiment of the present application also includes: generating occupied space response information based on the total number of occupied data blocks; and sending the occupied space response information to the scheduling device.

[0131] Further, optionally, generating occupied space response information based on the total number of occupied data blocks includes: obtaining the number of occupied data blocks from each data block management respectively; calculating the space occupied by the number of occupied data blocks of each data block management to obtain the total number of occupied data blocks; generating occupied space response information based on the total number of occupied data blocks.

[0132] Specifically, on the data processing device side, since all data block management (segment) calculations are parallel, when the Block Server has sufficient processing power, the time used for the entire calculation is roughly equivalent to the difference in data blocks of a single segment. The entire calculation time can be completed in seconds, which is several orders of magnitude shorter than the calculation time of the algorithms in the prior art.

[0133] Since the entire calculation does not require data block deduplication operations in memory, there is no additional memory usage requirement, which reduces the demand for system resources and the impact on the business.

[0134] Example 3

[0135] According to another aspect of an embodiment of the present invention, another data processing method is provided. On the scheduling device side, Figure 6 1 is a flow chart of a method for data processing according to Embodiment 3 of the present invention. Figure 6 As shown, including:

[0136] Step S602, receiving a snapshot space occupation request;

[0137] Step S604, generating at least one occupied space calculation task according to the snapshot occupied space request;

[0138] Step S606, sending each occupied space calculation task to a corresponding data processing device.

[0139] Optionally, the data processing method provided in the embodiment of the present application also includes: receiving occupied space response information sent by at least one data processing device; and counting the total number of occupied data blocks based on the occupied space response information.

[0140] Specifically, the data processing method provided in the embodiment of the present application is applied to the scheduling device in Example 1, wherein, after receiving the request to query the snapshot occupied space, the scheduling device calculates the snapshot pair that needs to be updated incrementally (the new parent snapshot S1 and the snapshot S2 that needs to be updated) according to the snapshot list input by the management; the Block Master calculates the number of snapshot segments (that is, the task of calculating the occupied space in the embodiment of the present application) and sends it to the corresponding Block server (that is, the data processing device in the embodiment of the present application) to complete the calculation of the data difference of each Segment; the Block Master waits for all Segments to be updated, summarizes the occupied space of all Segments, and completes the query request.

[0141] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the data processing method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0143] Example 4

[0144] According to an embodiment of the present invention, a device for implementing the above data processing method is also provided. Figure 7 As shown, Figure 7 is a schematic diagram of a data processing device according to a fourth embodiment of the present invention, the device comprising:

[0145] A receiving module 72 is used to receive a task for calculating occupied space; an obtaining module 74 is used to obtain the number of occupied data blocks of a specified snapshot in the task for calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot; a calculating module 76 is used to obtain the total number of occupied data blocks in the task for calculating occupied space based on the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots.

[0146] Example 5

[0147] According to an embodiment of the present invention, a device for implementing the above data processing method is also provided. Figure 8 As shown, Figure 8 is a schematic diagram of a data processing device according to a fifth embodiment of the present invention, the device comprising:

[0148] The receiving module 82 is used to receive a snapshot occupied space request; the task generating module 84 is used to generate at least one calculation occupied space task according to the snapshot occupied space request; the sending module 86 is used to send each calculation occupied space task to the corresponding data processing device.

[0149] Example 6

[0150] According to another aspect of an embodiment of the present invention, a storage medium is provided, the storage medium including a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the data processing method in the above-mentioned embodiments 2 and 3.

[0151] Example 7

[0152] According to yet another aspect of an embodiment of the present invention, a processor is provided, and the processor is used to run a program, wherein the data processing method in the above-mentioned embodiments 2 and 3 is executed when the program is run.

[0153] Example 8

[0154] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0155] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0156] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a task for calculating occupied space; obtaining the number of occupied data blocks of a specified snapshot in the task for calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot; obtaining the total number of occupied data blocks in the task for calculating occupied space based on the number of occupied data blocks and the number of difference data blocks; wherein the specified snapshot and the first parent snapshot are adjacent snapshots.

[0157] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot includes: obtaining the number of occupied data blocks of the specified snapshot from a preset index mapping table; determining whether the first parent snapshot of the specified snapshot exists; if the judgment result is yes, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot based on the status of the first parent snapshot in the preset index mapping table; if the judgment result is no, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks.

[0158] Further, optionally, in the present embodiment, the storage medium is configured to store program code for executing the following steps: based on the status of the first parent snapshot in the preset index mapping table, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot includes: if the first parent snapshot is not in the preset index mapping table, traversing the preset snapshot chain to obtain the second parent snapshot; obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot; if the first parent snapshot is in the preset index mapping table, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot.

[0159] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: acquiring the second parent snapshot includes: acquiring a snapshot whose distance from the designated snapshot is less than a preset distance; and determining the snapshot as the second parent snapshot.

[0160] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining the number of differential data blocks between the second parent snapshot and the specified snapshot includes: obtaining the number of differential data blocks between the second parent snapshot and the specified snapshot based on whether the snapshot identifiers managed by the data blocks in a preset index mapping table are the same; or, obtaining the number of differential data blocks between the second parent snapshot and the specified snapshot based on the fingerprint information of the data blocks.

[0161] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: updating difference data information in a preset index mapping table, wherein the difference data information is used to indicate the number of data blocks of a snapshot that is different from a specified snapshot.

[0162] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: generating occupied space response information according to the total number of occupied data blocks; and sending the occupied space response information to the scheduling device.

[0163] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: generating occupied space response information based on the total number of occupied data blocks includes: obtaining the number of occupied data blocks from each data block management respectively; calculating the space occupied by the number of occupied data blocks of each data block management to obtain the total number of occupied data blocks; generating occupied space response information based on the total number of occupied data blocks.

[0164] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0165] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0170] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing system, comprising: Scheduling device and data processing device, wherein, The scheduling device is used to receive a snapshot occupied space request; generate at least one calculation occupied space task according to the snapshot occupied space request, and send each calculation occupied space task to the corresponding data processing device; The data processing device is used to receive a task for calculating occupied space, obtain the number of occupied data blocks of a specified snapshot in the task for calculating occupied space, and the number of differential data blocks between the specified snapshot and a first parent snapshot of the specified snapshot; obtain the total number of occupied data blocks in the task for calculating occupied space according to the number of occupied data blocks and the number of differential data blocks, wherein the specified snapshot and the first parent snapshot are adjacent snapshots; Among them, obtaining the total number of occupied data blocks in the task of calculating occupied space based on the number of occupied data blocks and the number of difference data blocks includes: in response to the non-existence of the first parent snapshot, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks; in response to the existence of the first parent snapshot, obtaining the total number of occupied data blocks by superimposing the number of difference data blocks based on the snapshot chain.

2. A method for data processing, comprising: Receive a task to calculate occupied space; Obtaining the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and a first parent snapshot of the specified snapshot; According to the number of occupied data blocks and the number of difference data blocks, the total number of occupied data blocks in the task of calculating occupied space is obtained; The specified snapshot and the first parent snapshot are adjacent snapshots; Among them, obtaining the total number of occupied data blocks in the task of calculating occupied space based on the number of occupied data blocks and the number of difference data blocks includes: in response to the non-existence of the first parent snapshot, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks; in response to the existence of the first parent snapshot, obtaining the total number of occupied data blocks by superimposing the number of difference data blocks based on the snapshot chain.

3. The method according to claim 2, wherein: Obtaining the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and the first parent snapshot of the specified snapshot includes: Obtain the number of occupied data blocks of the specified snapshot from a preset index mapping table; Determine whether the first parent snapshot of the specified snapshot exists; If the judgment result is yes, the number of difference data blocks between the first parent snapshot and the designated snapshot is obtained according to the state of the first parent snapshot in the preset index mapping table.

4. The method according to claim 3, wherein: According to the state of the first parent snapshot in the preset index mapping table, obtaining the number of difference data blocks between the first parent snapshot and the specified snapshot includes: If the first parent snapshot is not in the preset index mapping table, traverse the preset snapshot chain to obtain the second parent snapshot; Obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot; If the first parent snapshot is in the preset index mapping table, the number of difference data blocks between the first parent snapshot and the specified snapshot is obtained.

5. The method according to claim 4, wherein: Getting the second parent snapshot includes: Acquire a snapshot whose distance from the specified snapshot is less than a preset distance; The snapshot is determined as the second parent snapshot.

6. The method according to claim 4, wherein: Obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot includes: Obtaining the number of difference data blocks between the second parent snapshot and the specified snapshot according to whether the snapshot identifiers of the data block management in the preset index mapping table are the same; or, The number of difference data blocks between the second parent snapshot and the designated snapshot is obtained according to the fingerprint information of the data block.

7. The method according to claim 4, wherein: The method further comprises: The difference data information in the preset index mapping table is updated, wherein the difference data information is used to indicate the number of data blocks of the snapshot that is different from the specified snapshot.

8. The method according to claim 2, characterized in that: The method further comprises: Generate occupied space response information according to the total number of occupied data blocks; The occupied space response information is sent to a scheduling device.

9. The method according to claim 8, characterized in that The generating of occupied space response information according to the total number of occupied data blocks includes: Obtain the number of occupied data blocks from each data block management; Calculate the space occupied by the number of occupied data blocks managed by each data block to obtain the total number of occupied data blocks; Generate occupied space response information according to the total number of occupied data blocks.

10. A method for data processing, comprising: Receive a snapshot space usage request; At least one computing occupied space task is generated according to the snapshot occupied space request, wherein the computing occupied space task includes the number of occupied data blocks of the specified snapshot and the number of differential data blocks between the specified snapshot and the first parent snapshot of the specified snapshot, the specified snapshot and the first parent snapshot are adjacent snapshots, and obtaining the total number of occupied data blocks in the computing occupied space task according to the number of occupied data blocks and the number of differential data blocks comprises: in response to the non-existence of the first parent snapshot, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks; in response to the existence of the first parent snapshot, obtaining the total number of occupied data blocks by superimposing the number of differential data blocks based on the snapshot chain; Each computing space occupation task is sent to a corresponding data processing device.

11. The method according to claim 10, characterized in that The method further comprises: receiving occupied space response information sent by at least one of the data processing devices; The total number of occupied data blocks is counted according to the occupied space response information.

12. A data processing device, comprising: A receiving module, used for receiving a task for calculating occupied space; An acquisition module, used to acquire the number of occupied data blocks of a specified snapshot in the task of calculating occupied space, and the number of difference data blocks between the specified snapshot and a first parent snapshot of the specified snapshot; A calculation module, used for obtaining the total number of occupied data blocks in the calculation occupied space task according to the number of occupied data blocks and the number of difference data blocks; The specified snapshot and the first parent snapshot are adjacent snapshots; The calculation module is further used for: in response to the non-existence of the first parent snapshot, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks; in response to the existence of the first parent snapshot, obtaining the total number of occupied data blocks by superimposing the number of difference data blocks based on the snapshot chain.

13. A data processing device, comprising: A receiving module, used for receiving a snapshot space occupation request; a task generation module, configured to generate at least one task for calculating occupied space according to the snapshot occupied space request, wherein the task for calculating occupied space includes the number of occupied data blocks of a specified snapshot, and the number of differential data blocks between the specified snapshot and a first parent snapshot of the specified snapshot, the specified snapshot and the first parent snapshot being adjacent snapshots, and obtaining the total number of occupied data blocks in the task for calculating occupied space according to the number of occupied data blocks and the number of differential data blocks comprises: in response to the non-existence of the first parent snapshot, determining the number of occupied data blocks of the specified snapshot as the total number of occupied data blocks; in response to the existence of the first parent snapshot, obtaining the total number of occupied data blocks by superimposing the number of differential data blocks based on the snapshot chain; The sending module is used to send each space occupation calculation task to the corresponding data processing device.

14. A storage medium, wherein: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 2 to 9.

15. A processor, wherein: The processor is used to run a program, wherein the program executes the data processing method according to any one of claims 2 to 9 when running.

Images