Data processing method and apparatus

Abstract

This specification provides a data processing method and apparatus. The data processing method includes: receiving a snapshot preparation instruction and a snapshot generation sequence number; if a data write instruction is determined to exist, obtaining a current data write sequence number corresponding to the data to be written carried in the data write instruction; writing the data to be written to a corresponding target storage device according to the current data write sequence number; and processing the data in the target storage device if the current data write sequence number matches the snapshot generation sequence number. By receiving the snapshot generation sequence number and obtaining the current data write sequence number corresponding to the data to be written during data writing, and processing the data in the target disk if the current data write sequence number is the snapshot generation sequence number, the data processing time is shortened because the sequence number lookup process is eliminated after obtaining the current data write sequence number.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method. Background Technology

[0002] More and more enterprises have migrated their businesses to cloud computing platforms, leading to a corresponding increase in the need for data backup in the cloud. For disk groups that hold critical data, users will develop periodic disk group snapshot plans to restore the data in the event of a disaster, ensuring the continuity of critical business operations.

[0003] In the existing disk group consistency snapshot algorithm of LSBD (log structured block device), snapshots are divided into three phases: prepare, query, and commit. In the prepare phase, the control terminal initiates prepare for each cloud disk, and all cloud disks enter the prepare phase. From this phase onwards, each write I / O operation initiated for a cloud disk obtains a globally monotonically increasing sequence number from the same sequence number generator to order the I / O. In the query phase, the current maximum sequence number of each cloud disk is queried and compared to determine the current globally maximum sequence number, which serves as the snapshot commit deadline. In the commit phase, a commit is initiated for each cloud disk using the maximum sequence number. At this point, the cloud disks stop retrieving sequence numbers, and the I / O sequence numbers since the start of prepare are traversed, comparing the I / O sequence number with the maximum sequence number. I / Os with a sequence number less than or equal to the maximum sequence number are included in the snapshot. However, the current disk group snapshots ensure consistency by obtaining a globally monotonically increasing sequence number from the number generator. As the number of cloud disks increases, the time for obtaining the number increases significantly, potentially reaching several seconds. The duration of snapshot creation time and IO spikes caused by numbering also increases accordingly. Summary of the Invention

[0004] In view of this, embodiments of this specification provide two data processing methods. One or more embodiments of this specification also relate to a data processing apparatus, a data processing system, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0006] Receive snapshot preparation instructions and snapshot generation sequence number;

[0007] If a data write instruction is determined to exist, the current data write sequence number corresponding to the data to be written carried in the data write instruction is obtained;

[0008] According to the current data write sequence number, the data to be written is written to the corresponding target storage device;

[0009] If the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device.

[0010] According to a second aspect of the embodiments of this specification, a data processing method is provided, applied to a data processing system, the system including a data storage module and a data control module, wherein the method includes:

[0011] Upon receiving a snapshot creation request, the data control module determines the target sequence number distribution node and sends the first data write sequence number to the target sequence number distribution node.

[0012] Send a snapshot preparation instruction and a snapshot generation sequence number to the data storage module, and send a second data writing sequence number to the target sequence number distribution node according to the first data writing sequence number;

[0013] Upon receiving a data write instruction, the data storage module obtains the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node. The current data write sequence number is either the first data write sequence number or the second data write sequence number.

[0014] Based on the current data write sequence number, the data to be written is written to the corresponding target storage device, and if the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device.

[0015] According to a third aspect of the embodiments of this specification, a data processing system is provided, including a data storage module and a data control module;

[0016] The data control module is configured to, upon receiving a snapshot creation request, determine the target sequence number distribution node and send a first data write sequence number to the target sequence number distribution node.

[0017] Send a snapshot preparation instruction and a snapshot generation sequence number to the data storage module, and send a second data writing sequence number to the target sequence number distribution node according to the first data writing sequence number;

[0018] The data storage module is configured to, upon receiving a data write instruction, obtain from the target sequence number distribution node the current data write sequence number corresponding to the data to be written carried in the data write instruction, wherein the current data write sequence number is either the first data write sequence number or the second data write sequence number.

[0019] Based on the current data write sequence number, the data to be written is written to the corresponding target storage device, and if the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device.

[0020] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0021] The data acquisition module is configured to receive data snapshot generation commands and snapshot generation sequence numbers;

[0022] The number retrieval module is configured to, when it is determined that there is a data write instruction, obtain the current data write sequence number corresponding to the data to be written carried in the data write instruction;

[0023] The data writing module is configured to write the data to be written to the corresponding target storage device according to the current data writing sequence number;

[0024] The determination module is configured to perform data processing on the data in the target storage device if it is determined that the current data write sequence number of the target storage device matches the snapshot generation sequence number.

[0025] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0026] Memory and processor;

[0027] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0028] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0029] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0030] This specification provides a data processing method and apparatus. The data processing method includes: receiving a snapshot preparation instruction and a snapshot generation sequence number; if a data write instruction is determined to exist, obtaining a current data write sequence number corresponding to the data to be written carried in the data write instruction; writing the data to be written to a corresponding target storage device according to the current data write sequence number; and processing the data in the target storage device if the current data write sequence number matches the snapshot generation sequence number. By receiving the snapshot generation sequence number and obtaining the current data write sequence number corresponding to the data to be written during data writing, and processing the data in the target disk if the current data write sequence number is the snapshot generation sequence number, the data processing time is shortened because the sequence number lookup process is eliminated after obtaining the current data write sequence number. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0032] Figure 2 This is a schematic diagram of a target storage device for a data processing method provided in one embodiment of this specification;

[0033] Figure 3 This is a snapshot diagram illustrating a data processing method provided in one embodiment of this specification;

[0034] Figure 4 This is a flowchart of another data processing method provided in one embodiment of this specification;

[0035] Figure 5 This is a schematic diagram of a data processing method provided in one embodiment of this specification;

[0036] Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0037] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0038] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0039] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0041] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0042] Preparation phase: Initiate prepare for each cloud disk, and all cloud disks enter the prepare phase.

[0043] Query phase: Query the current maximum sequence number of each cloud disk, compare it, and obtain the current global maximum sequence number as the snapshot submission deadline.

[0044] Submission phase: Use the maximum sequence number to initiate a commit to each cloud disk, at which point the cloud disk stops retrieving sequence numbers.

[0045] IO: Input / Output.

[0046] Virtual machine disk: Virtual machine block device.

[0047] Log-structured block device: A block device built based on a log structure.

[0048] Disk snapshot: Virtual machine disk block device snapshot.

[0049] VM: Virtual Machine.

[0050] Device: Virtual machine block device.

[0051] EBS: Elastic Block Service, a block device storage cluster.

[0052] LSBD: Log Structured Block Device.

[0053] Snapshot: A disk snapshot. A snapshot is a quick record of the current state of the disk. After a series of read and write operations are performed on the virtual machine, the snapshot can be used to restore the original state.

[0054] HardLink: Hard link technology used for sharing data among multiple files.

[0055] LBA: Logical Block Address.

[0056] Data file: A file that stores user data.

[0057] Index file: Stores the mapping relationship between user address space LBA and DataFile.

[0058] Segment: A logical segment of a block device. The entire virtual block device is logically divided into several segments.

[0059] RiverMaster: The node responsible for scheduling the entire snapshot process.

[0060] BlockMaster: The node responsible for managing the metadata of cloud disks and snapshots, as well as controlling the snapshot process within the cluster.

[0061] BlockServer: The node responsible for serving cloud disks and snapshot segments.

[0062] SequenceServer: The sequence number generator, responsible for sending sequence numbers to order write I / O.

[0063] In the existing disk group consistency snapshot algorithm of LSBD, snapshots are divided into three phases: prepare, query, and commit. In the prepare phase, the control terminal initiates prepare for each cloud disk, and all cloud disks enter the prepare phase. From this phase onwards, each write I / O operation initiated for a cloud disk obtains a globally monotonically increasing sequence number from the same sequence number generator to order the I / O. In the query phase, the current maximum sequence number of each cloud disk is queried and compared to determine the current globally maximum sequence number, which serves as the snapshot commit deadline. In the commit phase, a commit is initiated for each cloud disk using the maximum sequence number. At this point, the cloud disks stop retrieving sequence numbers, and the sequence numbers of each I / O operation since prepare began are iterated through, comparing the I / O sequence number with the maximum sequence number. I / Os with a sequence number less than or equal to the maximum sequence number are included in the snapshot.

[0064] The current solution takes time from receiving the prepare statement to receiving the commit statement. The more cloud disks in the disk group, the longer the total time (i.e., query phase time) is spent querying the maximum sequence number from all cloud disks, and the longer the time period for obtaining the sequence number is. This ultimately results in a longer disk group snapshot creation time and longer duration of IO spikes.

[0065] Based on this, two data processing methods are provided in this specification. This specification also relates to a data processing apparatus, a data processing system, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0066] See Figure 1 , Figure 1 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0067] Step 102: Receive the snapshot preparation instruction and the snapshot generation sequence number.

[0068] The snapshot preparation command can be understood as the command that determines the start of the preparation phase; the snapshot generation sequence number can be understood as the sequence number of the data to be processed, for example, the snapshot generation sequence number is 5.

[0069] In practical applications, when the control terminal (RiverMaster) receives a snapshot request, it sends a snapshot preparation command to the disk group server (BlockServer). Upon receiving the snapshot preparation command and the snapshot generation sequence number, the disk group server enters the preparation phase and records the snapshot generation sequence number, which can be N+1.

[0070] For example, receive the snapshot preparation instruction and snapshot generation sequence number from the master node, enter the preparation node, and record the snapshot generation sequence number as 5.

[0071] The embodiments in this specification improve data processing efficiency by generating sequence numbers through receiving snapshots during the preparation phase, eliminating the need to retrieve the carry number and query the maximum number after each IO write operation.

[0072] Step 104: If a data write instruction is determined to exist, obtain the current data write sequence number corresponding to the data to be written carried in the data write instruction.

[0073] Here, the data write instruction can be understood as the instruction for IO write operation, such as the instruction for the client to perform IO write operation; the data to be written is the data to be written by the IO write operation, such as log data; the current data write sequence number can be understood as the sequence number obtained when there is an IO write operation, such as obtaining the current data write sequence number as 5 when an IO write operation is received.

[0074] In practical applications, when a client performs an I / O write operation on the disk group server's hard drive, the disk group server obtains a current data write sequence number corresponding to that I / O write operation from the target sequence number distribution node (SequenceServer). Before the control terminal updates the write sequence number to the target sequence number distribution node, the current data write sequence number obtained for each I / O write operation can always be N. After the control terminal updates the write sequence number, it can obtain the current data write sequence number N+1.

[0075] It should be noted that when the snapshot generation sequence number is N+1, the current data write sequence number can also be any number different from N+1. For example, if the snapshot generation sequence number is 5, the current write sequence number can be 4, or it can be 6. This specification does not impose any limitations on the embodiments.

[0076] For example, upon receiving a snapshot preparation instruction and a snapshot generation sequence number, the system enters the preparation phase. Whenever a new IO write operation occurs, a sequence number is retrieved. If IO write operation w1 is received, sequence number 4 is obtained from the target sequence number distribution node; if IO write operation w2 is received, sequence number 4 is obtained from the target sequence number distribution node; and if IO write operation w3 is received, sequence number 4 is obtained from the target sequence number distribution node. Therefore, the current data write sequence number corresponding to IO write operation w1 is 4, the current data write sequence number corresponding to IO write operation w2 is 4, and the current data write sequence number corresponding to IO write operation w3 is 4.

[0077] In one possible implementation, obtaining the current data write sequence number corresponding to the data to be written carried in the data write instruction includes:

[0078] Based on the node information carried in the snapshot preparation command, determine the target sequence number distribution node;

[0079] Obtain the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node.

[0080] In this context, node information can be understood as the path information of a node in a distributed system, such as the node's IP address; the target sequence number distribution node can be understood as the node that distributes and writes the sequence number.

[0081] In practical applications, the control terminal needs to inform the disk group server of the node information of the target sequence number distribution node so that the disk group server knows which node to retrieve the number from, and thus retrieves the number from the corresponding target sequence number distribution node when there is an IO write operation.

[0082] For example, in the case of receiving a snapshot preparation command, obtain the node information of the target sequence number distribution node A from the snapshot preparation command, so that the corresponding target sequence number distribution node A can be found through the node information, and the sequence number can be obtained from the target sequence number distribution node A.

[0083] Embodiments of this specification obtain the current data write sequence number, so as to determine whether to take a snapshot subsequently according to the current data write sequence number.

[0084] Step 106: Write the data to be written into the corresponding target storage device according to the current data write sequence number.

[0085] Among them, writing the data to be written into the corresponding target storage device according to the current data write sequence number can be understood as writing the data to be written into the corresponding target storage device after obtaining the current write sequence number; the target storage device can be understood as an LSBD disk group, a segment, etc.

[0086] In practical applications, after obtaining the current data write sequence number corresponding to the IO write operation, the IO write operation can be executed. In the case of executing the IO write operation, there is a corresponding write order. For example, there are IO write operation w1 and IO write operation w2, w1_complete represents the completion event of w1, that is, the write is successful, w2_start represents the initiation event of w2, and w1_complete < w2_start, then it is defined as w1 < w2. That is to say, after the IO write operation w1 is completed, the IO write operation w2 can be carried out.

[0087] For example, after the IO write operation w1 receives the current write sequence number 5, write the data to be written corresponding to the IO write operation w1 into the corresponding target storage device.

[0088] In a realizable manner, after writing the data to be written into the corresponding target storage device according to the current data write sequence number, it further includes:

[0089] Determine the data storage location of the data to be written in the target storage device;

[0090] Generate index data of the data to be written according to the data storage location;

[0091] Write the index data of the data to be written into the target storage device.

[0092] Here, the data storage location can be understood as the logical disk address of the data. For example, the logical disk address could be LBA1:<datafile1,offset1,length1> Where LBA1 is the logical disk address, meaning that LBA1 is located in datafile1, with an offset of offset1 relative to the start position of datafile1 and a length of length1; the index data can be the data that records the data storage location to be written, i.e., the index file.

[0093] In practical applications, see Figure 2 , Figure 2 The system includes an LSBD client and an LSBD device. The LSBD device includes data file segments 0 to N, totaling N+1 data file segments. The data to be written can be data file 1 to data file x in data file segment 0. After writing the data to be written, index data of the data to be written needs to be generated, namely index file 1 to index file x in data file segment 0, so that the corresponding data can be found later and the corresponding data can be processed.

[0094] For example, after IO write operation w1 receives the current write sequence number 5, it writes the data to be written corresponding to IO write operation w1 to the corresponding target storage device, and records the position of the data corresponding to IO write operation w1 in the target storage device: LBA1:<datafile1,offset1,length1> According to LBA1:<datafile1,offset1,length1> Generate index data (index file) and store the index data in the target storage device.

[0095] The embodiments in this specification describe the data written for IO write operations according to the sequence number, which facilitates the subsequent determination of snapshot data.

[0096] Step 108: If the current data write sequence number matches the snapshot generation sequence number, perform data processing on the data in the target storage device.

[0097] The phrase "the current data writing sequence number matches the snapshot generation sequence number" can be understood as "the current data writing sequence number is the same as the snapshot generation sequence number"; "data processing" can be understood as taking a snapshot or backing up the data.

[0098] In practical applications, the control terminal sends the snapshot generation sequence number to the target sequence number distribution node. When there is a new IO write operation, the disk group server retrieves the number from the target sequence number distribution node and obtains the new snapshot generation sequence number, and then processes the data immediately.

[0099] For example, if the snapshot generation sequence number is 5, upon receiving the snapshot preparation instruction and the snapshot generation sequence number, the system enters the preparation phase. Whenever a new IO write operation occurs, a sequence number is retrieved. For IO write operation w1, sequence number 4 is obtained from the target sequence number distribution node; for IO write operation w2, sequence number 4 is obtained from the target sequence number distribution node; and for IO write operation w3, sequence number 4 is obtained from the target sequence number distribution node. Therefore, the current data write sequence number corresponding to IO write operation w1 is 4, the current data write sequence number corresponding to IO write operation w2 is 4, the current data write sequence number corresponding to IO write operation w3 is 4, and the current data write sequence number corresponding to IO write operation w4 is 5. The current data write sequence number is then matched with the snapshot generation sequence number, and data processing is performed on the data in the target storage device.

[0100] In one possible implementation, the data processing of the data in the target storage device includes:

[0101] Determine the historical data write sequence number in the target storage device that is different from the current data write sequence number;

[0102] Determine the historical backup data and the corresponding backup data write sequence number;

[0103] The write sequence number of the data to be backed up is determined based on the historical data write sequence number and the backup data write sequence number.

[0104] The target backup data is determined based on the sequence number of the data to be backed up, and a snapshot is performed on the target backup data.

[0105] The historical data write sequence number can be a sequence number prior to the most recent number retrieval. For example, if the most recent number retrieval was 5, then the historical data write sequence numbers could be 1, 2, 3, or 4. Historical backup data can be understood as data that has already been snapshotted. Historical backup data can be identified by backup identifiers or by comparing the backup data with the data in the target storage device. For example, if the data corresponding to historical data write sequence numbers 1, 2, and 3 had backup identifiers before this snapshot was taken, then the data corresponding to these three historical data write sequence numbers has been snapshotted and is called historical backup data. The data to be backed up write sequence number can be understood as the data write sequence number corresponding to the data to be snapshotted. For example, historical data write sequence numbers 1, 2, and 3 are the data to be backed up write sequence numbers.

[0106] In practical applications, data snapshot processing can be performed as incremental snapshots, which means excluding data that has been previously snapshotted from the current snapshot.

[0107] For example, if the snapshot generation sequence number is 5, upon receiving the snapshot preparation instruction and the snapshot generation sequence number, the system enters the preparation phase. Whenever a new IO write operation occurs, a sequence number is retrieved. For IO write operation w1, sequence number 4 is obtained from the target sequence number distribution node; for IO write operation w2, sequence number 4 is obtained from the target sequence number distribution node; for IO write operation w3, sequence number 4 is obtained from the target sequence number distribution node; and for IO write operation w4, sequence number 5 is obtained from the target sequence number distribution node. Therefore, the current data write sequence number corresponding to IO write operation w1 is 4, the current data write sequence number corresponding to IO write operation w2 is 4, the current data write sequence number corresponding to IO write operation w3 is 4, and the current data write sequence number corresponding to IO write operation w4 is 5. The current data write sequence number is matched with the snapshot generation sequence number, and data processing is performed on the data in the target storage device. At this time, the current data write sequence number is 5, so the historical backup data is determined. According to the backup identifier, the historical backup data can be found to be the data corresponding to IO write operation w1 and IO write operation w2. The data corresponding to IO write operation w1 and IO write operation w2 are excluded, and snapshot processing is performed on the data corresponding to IO write operation w3.

[0108] For example, if the snapshot generation sequence number is 5, upon receiving the snapshot preparation instruction and the snapshot generation sequence number, the system enters the preparation phase. Whenever a new IO write operation occurs, a sequence number is retrieved. If IO write operation w1 is received, sequence number 2 is retrieved from the target sequence number distribution node; if IO write operation w2 is received, sequence number 3 is retrieved; if IO write operation w3 is received, sequence number 4 is retrieved; and if IO write operation w4 is received, sequence number 5 is retrieved. Therefore, the current data write sequence number corresponding to IO write operation w1 is 2, the current data write sequence number corresponding to IO write operation w2 is 3, the current data write sequence number corresponding to IO write operation w3 is 4, and the current data write sequence number corresponding to IO write operation w4 is 5. The current data write sequence number is matched with the snapshot generation sequence number to process the data in the target storage device. At this time, the current data write sequence number is 5, so the historical backup data is determined. By comparing the backup data with the data in the target storage device, the historical backup data corresponding to IO write operations w1 and w2 can be found. The data corresponding to IO write operations w1 and w2 are excluded, and the data corresponding to IO write operation w3 is snapshotted.

[0109] In one possible implementation, the data processing of the data in the target storage device includes:

[0110] Determine the historical data write sequence number in the target storage device that is different from the current data write sequence number;

[0111] Based on the historical data write sequence number, the target backup data is determined, and a snapshot is performed on the target backup data.

[0112] In practical applications, data snapshot processing can be performed as a full snapshot, which means that data that has been previously snapshotted is also snapshotted.

[0113] For example, if the snapshot generation sequence number is 5, upon receiving the snapshot preparation instruction and the snapshot generation sequence number, the system enters the preparation phase. Whenever a new IO write operation occurs, a sequence number is retrieved. For IO write operation w1, sequence number 4 is obtained from the target sequence number distribution node; for IO write operation w2, sequence number 4 is obtained from the target sequence number distribution node; for IO write operation w3, sequence number 4 is obtained from the target sequence number distribution node; and for IO write operation w4, sequence number 5 is obtained from the target sequence number distribution node. Therefore, the current data write sequence number corresponding to IO write operation w1 is 4, the current data write sequence number corresponding to IO write operation w2 is 4, the current data write sequence number corresponding to IO write operation w3 is 4, and the current data write sequence number corresponding to IO write operation w4 is 5. The current data write sequence number matches the snapshot generation sequence number, and data processing is performed on the data in the target storage device. Since the current data write sequence number is 5, snapshot processing is performed on the data corresponding to IO write operations w1, w2, and w3.

[0114] In one possible implementation, the snapshot processing of the target backup data includes:

[0115] Determine the storage location of the target backup data in the target storage device;

[0116] The target backup data is copied from the storage location of the target storage device to the backup storage device, and the location information of the data to be backed up in the backup storage device is determined;

[0117] Generate index data for the target backup data based on the location information;

[0118] Write the index data of the target backup data into the backup storage device.

[0119] The backup storage device can be a device for taking snapshots; the target backup data can be the data to be snapshotted.

[0120] In practical applications, see Figure 3 , Figure 3 This includes an LSBD device and an LSBD snapshot. The LSBD device includes data file segments 0 to N, totaling N+1 data file segments. The LSBD snapshot also includes data file segments 0 to N, totaling N+1 data file segments. The target storage device can be an LSBD device, and the backup storage device can be an LSBD snapshot. Specifically, the storage location of the data to be snapshotted can be obtained from the target storage device (e.g., the location of data file 1 can be obtained based on index file 1). The target backup data can be obtained from the storage location and copied to the LSBD snapshot. After the copying is completed, the location of the snapshotted data in the LSBD snapshot is obtained. Then, a new index file is generated based on the location of the data in the LSBD snapshot, and the index file is written to the LSBD snapshot.

[0121] It should be noted that in backup storage devices, there can be only one index file, meaning that multiple data files correspond to one index file, which improves data retrieval efficiency.

[0122] For example, to perform snapshot processing on the data corresponding to IO write operation w3, the storage location of the data corresponding to IO write operation w3 is obtained from the target storage device, the data corresponding to IO write operation w3 is obtained from the storage location, and the data corresponding to IO write operation w3 is copied to the backup storage device where the snapshot is taken. After the copy is completed, the location of the data corresponding to IO write operation w3 in the backup storage device is obtained, and then a new index file is generated based on the location of the data corresponding to IO write operation w3 in the backup storage device, and the index file is written to the backup storage device.

[0123] In one possible implementation, after writing the data to be written to the corresponding target storage device according to the current data write sequence number, the method further includes:

[0124] If it is determined that the current data write sequence number does not match the snapshot generation sequence number, data processing is performed on the data in the target storage device in response to the snapshot confirmation command.

[0125] The snapshot confirmation instruction can be understood as an instruction to confirm the result of the snapshot, which is the instruction that begins the commit phase.

[0126] In practical applications, the control terminal sends a commit command to all disk group nodes where the segment is located to confirm that all segments have received sequence number N+1 and have performed a snapshot. If not, it means that there are no new IO write operations at the moment, so the numbering is stopped immediately, and the snapshot is executed, and the disk group snapshot is created.

[0127] This specification provides a data processing method and apparatus. The data processing method includes: receiving a snapshot preparation instruction and a snapshot generation sequence number; if a data write instruction is determined to exist, obtaining a current data write sequence number corresponding to the data to be written carried in the data write instruction; writing the data to be written to a corresponding target storage device according to the current data write sequence number; and processing the data in the target storage device if the current data write sequence number matches the snapshot generation sequence number. By receiving the snapshot generation sequence number and obtaining the current data write sequence number corresponding to the data to be written during data writing, and processing the data in the target disk if the current data write sequence number is the snapshot generation sequence number, the data processing time is shortened because the sequence number lookup process is eliminated after obtaining the current data write sequence number.

[0128] See Figure 4 This specification also provides a data processing method for use in a data processing system, the system including a data storage module and a data control module, wherein the method includes:

[0129] Step 402: Upon receiving a snapshot creation request, the data control module determines the target sequence number distribution node and sends the first data write sequence number to the target sequence number distribution node.

[0130] Send a snapshot preparation instruction and a snapshot generation sequence number to the data storage module, and send a second data writing sequence number to the target sequence number distribution node according to the first data writing sequence number;

[0131] Step 404: Upon receiving a data write instruction, the data storage module obtains the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node, wherein the current data write sequence number is either the first data write sequence number or the second data write sequence number.

[0132] Based on the current data write sequence number, the data to be written is written to the corresponding target storage device, and if the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device.

[0133] The data control module can be RiverMaster, i.e., the control terminal; the data storage module can include BlockMaster and BlockServer.

[0134] In one possible implementation, the data storage module includes a master node and child nodes;

[0135] Accordingly, upon receiving a data write instruction, obtaining the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node includes:

[0136] The master node receives the snapshot preparation command and snapshot generation sequence number sent by the data control module, and forwards the snapshot preparation command and snapshot generation sequence number to the child nodes;

[0137] Upon receiving a data write instruction, the child node obtains the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node.

[0138] Based on the current data write sequence number, the data to be written is written to the corresponding target storage device, and if the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device.

[0139] The master node can be BlockMaster, and the child nodes can be BlockServer.

[0140] In practical applications, see Figure 5The control end is responsible for scheduling the overall snapshot process, the master node is responsible for managing device metadata and snapshot metadata as well as controlling the snapshot process within the cluster, the child nodes are responsible for serving the storage devices, the sequence number distribution node is responsible for sending sequence numbers to order write IO, the client represents the IO client of the LSBD device (such as VM), the management end represents the initiator of the snapshot creation operation, cluster 0 is the cluster where the data is located, and cluster 1 is the cluster where the snapshot data is located. The management terminal initiates a request to the control terminal to create a disk group snapshot. The control terminal selects a sequence number distribution node and sends it a sequence number N. All subsequent child nodes that retrieve sequence numbers return N. The control terminal sends a preparation command and a maximum sequence number N+1 to the master node. The master node sends a preparation command and a maximum sequence number N+1 to all child nodes containing data file segments, requiring all write IOs entering the data file segment to retrieve sequence numbers, and using sequence number N+1 as the snapshot cutoff point. Once N+1 is received, the retrieval process immediately stops, and the snapshot is executed, generating an index file for the snapshot data file segment. The client continuously writes IOs to the data file segments in the storage device (e.g., cloud disks). For all write IOs to the relevant cloud disks, the child nodes retrieve sequence numbers from the sequence number distribution node, which returns N. The control terminal sends the snapshot cutoff point N+1 to the sequence number distribution node. All subsequent child nodes that retrieve sequence numbers return N+1, and the child nodes then execute the snapshot if they retrieve N+1.

[0141] In one possible implementation, after processing the data in the target storage device, the method further includes:

[0142] The data control module sends a snapshot confirmation command to the data storage module;

[0143] Upon receiving the snapshot confirmation instruction, the data storage module stops obtaining the current data write sequence number from the target sequence number distribution node, and returns snapshot result information to the data control module after the data write instruction is executed.

[0144] The data control module determines the data processing result of the data storage module on the target storage device based on the snapshot result information.

[0145] For example, the control terminal sends an acknowledgment command to the master node; the master node sends an acknowledgment command to the child nodes where the data file segments of all storage devices are located, confirming that all data file segments of all storage devices have received sequence number N+1 and have taken a snapshot.

[0146] In one possible implementation, after sending the confirmation instruction to the data storage module, the method further includes:

[0147] The data storage module processes the data in the target storage device when it determines that the current data write sequence number does not match the snapshot generation sequence number and receives the snapshot confirmation instruction.

[0148] For example, if a data file segment on a storage device is not snapshotted, it means there are no new write I / O operations at the moment. In this case, the number retrieval is stopped immediately, and the snapshot is executed, thus completing the disk group snapshot creation.

[0149] This specification provides a data processing method and apparatus. The data processing method includes: receiving a snapshot preparation instruction and a snapshot generation sequence number; if a data write instruction is determined to exist, obtaining a current data write sequence number corresponding to the data to be written carried in the data write instruction; writing the data to be written to a corresponding target storage device according to the current data write sequence number; and processing the data in the target storage device if the current data write sequence number matches the snapshot generation sequence number. By receiving the snapshot generation sequence number and obtaining the current data write sequence number corresponding to the data to be written during data writing, and processing the data in the target disk if the current data write sequence number is the snapshot generation sequence number, the data processing time is shortened because the sequence number lookup process is eliminated after obtaining the current data write sequence number.

[0150] This specification also provides a data processing system, including a data storage module and a data control module;

[0151] The data control module is configured to, upon receiving a snapshot creation request, determine the target sequence number distribution node and send a first data write sequence number to the target sequence number distribution node.

[0152] Send a snapshot preparation instruction and a snapshot generation sequence number to the data storage module, and send a second data writing sequence number to the target sequence number distribution node according to the first data writing sequence number;

[0153] The data storage module is configured to, upon receiving a data write instruction, obtain from the target sequence number distribution node the current data write sequence number corresponding to the data to be written carried in the data write instruction, wherein the current data write sequence number is either the first data write sequence number or the second data write sequence number.

[0154] Based on the current data write sequence number, the data to be written is written to the corresponding target storage device, and if the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device.

[0155] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 6 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0156] The data acquisition module 602 is configured to receive a data snapshot generation command and a snapshot generation sequence number;

[0157] The number retrieval module 604 is configured to, when it is determined that there is a data write instruction, obtain the current data write sequence number corresponding to the data to be written carried in the data write instruction;

[0158] The data writing module 606 is configured to write the data to be written to the corresponding target storage device according to the current data writing sequence number;

[0159] The determination module 608 is configured to perform data processing on the data in the target storage device when it is determined that the current data write sequence number of the target storage device matches the snapshot generation sequence number.

[0160] In one possible implementation, the number retrieval module 604 is also configured as follows:

[0161] Based on the node information carried in the snapshot preparation command, determine the target sequence number distribution node;

[0162] Obtain the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node.

[0163] In one possible implementation, the data writing module 606 is also configured as follows:

[0164] Determine the data storage location of the data to be written in the target storage device;

[0165] Based on the data storage location, generate index data for the data to be written;

[0166] Write the index data of the data to be written to the target storage device.

[0167] In one possible implementation, module 608 is also configured as follows:

[0168] Determine the historical data write sequence number in the target storage device that is different from the current data write sequence number;

[0169] Determine the historical backup data and the corresponding backup data write sequence number;

[0170] The write sequence number of the data to be backed up is determined based on the historical data write sequence number and the backup data write sequence number.

[0171] The target backup data is determined based on the sequence number of the data to be backed up, and a snapshot is performed on the target backup data.

[0172] In one possible implementation, module 608 is also configured as follows:

[0173] Determine the historical data write sequence number in the target storage device that is different from the current data write sequence number;

[0174] Based on the historical data write sequence number, the target backup data is determined, and a snapshot is performed on the target backup data.

[0175] In one possible implementation, module 608 is also configured as follows:

[0176] Determine the storage location of the target backup data in the target storage device;

[0177] The target backup data is copied from the storage location of the target storage device to the backup storage device, and the location information of the data to be backed up in the backup storage device is determined;

[0178] Generate index data for the target backup data based on the location information;

[0179] Write the index data of the target backup data into the backup storage device.

[0180] In one possible implementation, module 608 is also configured as follows:

[0181] If it is determined that the current data write sequence number does not match the snapshot generation sequence number, data processing is performed on the data in the target storage device in response to the snapshot confirmation command.

[0182] This specification provides a data processing method and apparatus. The data processing apparatus receives a snapshot preparation instruction and a snapshot generation sequence number. If a data write instruction is determined to exist, it obtains a current data write sequence number corresponding to the data to be written carried in the data write instruction. Based on the current data write sequence number, it writes the data to be written to a corresponding target storage device. If the current data write sequence number matches the snapshot generation sequence number, it processes the data in the target storage device. By receiving the snapshot generation sequence number and obtaining the current data write sequence number corresponding to the data to be written during data writing, and processing the data in the target disk when the current data write sequence number is the snapshot generation sequence number, the data processing time is shortened. Since the sequence number lookup process is eliminated after obtaining the current data write sequence number, the data processing time is shortened.

[0183] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0184] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0185] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0186] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0187] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 700 can also be a mobile or stationary server.

[0188] The processor 720 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0189] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0190] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0191] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data processing method described above.

[0192] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0193] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0194] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0195] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0196] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0197] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0198] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Receive snapshot preparation instructions and snapshot generation sequence number; If a data write instruction is confirmed, a target sequence number distribution node is determined based on the node information carried in the snapshot preparation instruction. The current data write sequence number corresponding to the data to be written carried in the data write instruction is obtained from the target sequence number distribution node. After the control terminal sends the updated write sequence number to the target sequence number distribution node, the current data write sequence number is updated to the updated write sequence number. According to the current data write sequence number, the data to be written is written to the corresponding target storage device; If the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device. Here, "the current data write sequence number matches the snapshot generation sequence number" means that the current data write sequence number and the snapshot generation sequence number are the same. "Data processing" means taking a snapshot or backing up the data.

2. The method according to claim 1, further comprising, after writing the data to be written to the corresponding target storage device according to the current data write sequence number: Determine the data storage location of the data to be written in the target storage device; Based on the data storage location, generate index data for the data to be written; Write the index data of the data to be written to the target storage device.

3. The method according to claim 1, wherein the data processing of the data in the target storage device includes: Determine the historical data write sequence number in the target storage device that is different from the current data write sequence number; Determine the historical backup data and the corresponding backup data write sequence number; The write sequence number of the data to be backed up is determined based on the historical data write sequence number and the backup data write sequence number. The target backup data is determined based on the sequence number of the data to be backed up, and a snapshot is performed on the target backup data.

4. The method according to claim 1, wherein the data processing of the data in the target storage device includes: Determine the historical data write sequence number in the target storage device that is different from the current data write sequence number; Based on the historical data write sequence number, the target backup data is determined, and a snapshot is performed on the target backup data.

5. The method according to claim 3 or 4, wherein the snapshot processing of the target backup data includes: Determine the storage location of the target backup data in the target storage device; The target backup data is copied from the storage location of the target storage device to the backup storage device, and the location information of the data to be backed up in the backup storage device is determined; Generate index data for the target backup data based on the location information; Write the index data of the target backup data into the backup storage device.

6. The method according to claim 1, further comprising, after writing the data to be written to the corresponding target storage device according to the current data write sequence number: If it is determined that the current data write sequence number does not match the snapshot generation sequence number, data processing is performed on the data in the target storage device in response to the snapshot confirmation command.

7. A data processing method applied to a data processing system, the system comprising a data storage module and a data control module, wherein, The method includes: Upon receiving a snapshot creation request, the data control module determines the target sequence number distribution node and sends the first data write sequence number to the target sequence number distribution node. Send a snapshot preparation instruction and a snapshot generation sequence number to the data storage module, and send a second data writing sequence number to the target sequence number distribution node according to the first data writing sequence number; Upon receiving a data write instruction, the data storage module determines a target sequence number distribution node based on the node information carried in the snapshot preparation instruction. It then obtains from the target sequence number distribution node the current data write sequence number corresponding to the data to be written carried in the data write instruction. This current data write sequence number is either the first data write sequence number or the second data write sequence number. After the control terminal sends an updated write sequence number to the target sequence number distribution node, the current data write sequence number is updated to the updated write sequence number. Based on the current data write sequence number, the data to be written is written to the corresponding target storage device. If the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device. Here, "the current data write sequence number matches the snapshot generation sequence number" means that the current data write sequence number and the snapshot generation sequence number are the same. "Data processing" means taking a snapshot or backing up the data.

8. The method according to claim 7, wherein the data storage module comprises a master node and a child node; Accordingly, upon receiving a data write instruction, obtaining the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node includes: The master node receives the snapshot preparation instruction and snapshot generation sequence number sent by the data control module, and forwards the snapshot preparation instruction and snapshot generation sequence number to the child nodes; Upon receiving a data write instruction, the child node obtains the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node. Based on the current data write sequence number, the data to be written is written to the corresponding target storage device, and if the current data write sequence number matches the snapshot generation sequence number, the data in the target storage device is processed.

9. The method according to claim 7, further comprising, after processing the data in the target storage device: The data control module sends a snapshot confirmation command to the data storage module; Upon receiving the snapshot confirmation instruction, the data storage module stops obtaining the current data write sequence number from the target sequence number distribution node, and returns snapshot result information to the data control module after the data write instruction is executed. The data control module determines the data processing result of the data storage module on the target storage device based on the snapshot result information.

10. A data processing system, comprising a data storage module and a data control module; The data control module is configured to, upon receiving a snapshot creation request, determine the target sequence number distribution node and send a first data write sequence number to the target sequence number distribution node. Send a snapshot preparation instruction and a snapshot generation sequence number to the data storage module, and send a second data writing sequence number to the target sequence number distribution node according to the first data writing sequence number; The data storage module is configured to, upon receiving a data write instruction, determine a target sequence number distribution node based on the node information carried in the snapshot preparation instruction, and obtain from the target sequence number distribution node the current data write sequence number corresponding to the data to be written carried in the data write instruction, wherein... The current data write sequence number is either the first data write sequence number or the second data write sequence number. After the control terminal sends the updated write sequence number to the target sequence number distribution node, the current data write sequence number is updated to the updated write sequence number. Based on the current data write sequence number, the data to be written is written to the corresponding target storage device. If the current data write sequence number matches the snapshot generation sequence number, data processing is performed on the data in the target storage device. Here, "the current data write sequence number matches the snapshot generation sequence number" means that the current data write sequence number and the snapshot generation sequence number are the same. "Data processing" means taking a snapshot or backing up the data.

11. A data processing apparatus, comprising: The data acquisition module is configured to receive snapshot preparation instructions and snapshot generation sequence numbers; The number retrieval module is configured to, when a data write instruction is determined, determine a target sequence number distribution node based on the node information carried in the snapshot preparation instruction, and obtain the current data write sequence number corresponding to the data to be written carried in the data write instruction from the target sequence number distribution node. The current data write sequence number is updated to the updated write sequence number after the control terminal sends the updated write sequence number to the target sequence number distribution node. The data writing module is configured to write the data to be written to the corresponding target storage device according to the current data writing sequence number; The determination module is configured to perform data processing on the data in the target storage device when it is determined that the current data write sequence number of the target storage device matches the snapshot generation sequence number. The current data write sequence number and the snapshot generation sequence number match means that the current data write sequence number and the snapshot generation sequence number are the same. The data processing means taking a snapshot or backing up the data.

12. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 6.

13. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 6.

Images