Data processing method and apparatus, and electronic device

Abstract

Embodiments of the present application provide a data processing method and device and electronic equipment. In the embodiments of the present application, when a creation instruction of any logical storage space is detected, or when it is determined to start creating a snapshot of any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node. Based on this, when any data update operation occurs on any data storage block on the node, frequent interaction with other nodes such as a metadata node is not required, but the block usage state of the data storage block in the locally stored first object mapping table is directly updated on the node based on the data update operation. In this way, without additional cross-node interaction, the efficiency of data processing is improved, the consistency of data update and block usage state update of the data storage block is ensured, and the performance of data processing is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device and electronic equipment. BACKGROUND

[0002] In a distributed storage system, data storage can be performed by using object storage, block storage, or file system storage, etc. Among them, the available storage space provided by the distributed storage system is divided into at least one logical storage space, and each logical storage space is further divided into a plurality of data storage blocks for data storage; and the data storage blocks of each logical storage space can be distributed on a plurality of different storage nodes.

[0003] In actual application of data processing by using any logical storage space, the block usage state (such as no data, data, and data to be deleted, etc.) of the data storage blocks of the logical storage space needs to be recorded as metadata information of the logical storage space to the metadata node; based on this, while updating the data in the logical storage space, the block usage state of the data storage blocks of the logical storage space on the metadata node also needs to be updated as much as possible to ensure the consistency of the data. However, since the data storage blocks of a logical storage space can be distributed on a plurality of storage nodes, when data update (such as data writing, etc.) is performed on the logical storage space, multiple storage nodes can be involved, so when the corresponding data update operations are performed on the involved storage nodes concurrently, only the metadata node can be interacted with sequentially to update the block usage state of the data storage blocks of the logical storage space, and the consistency of data update and block usage state update of the data storage blocks cannot be guaranteed, thereby affecting the performance of data processing. SUMMARY

[0004] Therefore, the present application provides a data processing method and device and electronic equipment to ensure the consistency of data update and block usage state update of the data storage blocks of the logical storage space, and improve the performance of data processing.

[0005] The present application provides a data processing method, which is applied to any storage node in a distributed storage system; the method comprises:

[0006] When a creation instruction of using at least one data storage block on the node as a data storage block of any logical storage space to be created is detected, or when it is determined to start a snapshot creation of any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate a block usage state of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table; and the specified time period corresponding to any object mapping table is determined according to a creation time of the object mapping table.

[0007] When any data update operation is detected on any data storage block on the node, the block usage state of the data storage block in the first object mapping table stored locally is updated based on the data update operation; the first object mapping table refers to an object mapping table with the latest creation time among object mapping tables corresponding to a logical storage space to which the data storage block belongs.

[0008] Embodiments of the present application further provide a data processing apparatus, which is applied to any storage node in a distributed storage system; the apparatus comprises:

[0009] The creation module is configured to, when a creation instruction of using at least one data storage block on the node as a data storage block of any logical storage space to be created is detected, or when it is determined to start a snapshot creation of any logical storage space on the node, create a new object mapping table for the logical storage space, and store the new object mapping table locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate a block usage state of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table; and the specified time period corresponding to any object mapping table is determined according to a creation time of the object mapping table.

[0010] The update module is configured to, when any data update operation is detected on any data storage block on the node, update the block usage state of the data storage block in the first object mapping table stored locally based on the data update operation; the first object mapping table refers to an object mapping table with the latest creation time among object mapping tables corresponding to a logical storage space to which the data storage block belongs.

[0011] Embodiments of the present application further provide an electronic device, which comprises:

[0012] a processor; and

[0013] A computer readable storage medium has computer program instructions stored therein, and the computer program instructions, when executed by a processor, cause the processor to perform the steps of the above method.

[0014] The application further provides a computer readable storage medium, which stores computer program instructions. When the computer program instructions are run by a processor, the processor executes the steps in the method.

[0015] From the above technical solution, it can be seen that in the embodiments of the application, when the creation indication of using at least one data storage block on the node as the data storage block of any logical storage space to be created is detected, or when it is determined to start creating a snapshot of any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node. Based on this, when any data update operation occurs on any data storage block on the node, frequent interaction with other nodes such as a metadata node is not required, but the block usage state of the data storage block in the locally stored first object mapping table can be directly updated on the node based on the data update operation. In this way, without additional cross-node interaction, the efficiency of data processing is improved, the consistency of data update and block usage state update of the data storage block is ensured, and the performance of data processing is improved.

[0016] In addition, in the embodiments of the application, the object mapping table used to indicate the block usage state of the data storage block on any storage node is stored locally on the storage node, and does not need to be synchronized to other nodes such as a metadata node for storage. In this way, when a fault occurs on the metadata node or one of the storage nodes and causes data loss, the data such as the object mapping table on the other storage nodes will not be affected by the fault node, thereby reducing the range of data pollution. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0018] Figure 1 A method flowchart is provided for the embodiments of the application.

[0019] Figure 2 Another method flowchart is provided for the embodiments of the application.

[0020] Figure 3 Still another method flowchart is provided for the embodiments of the application.

[0021] Figure 4 An apparatus structure schematic diagram is provided for the embodiments of the application.

[0022] Figure 5 An electronic device structure schematic diagram is provided for the embodiments of the application. DETAILED DESCRIPTION

[0023] In order to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the drawings.

[0024] Referring to Figure 1 , Figure 1 The method flowchart provided by the embodiments of the present application. The method is applied to any storage node in a distributed storage system. In the present embodiment, the storage node can be an electronic device such as a server, and the present embodiment does not specifically limit this. It should be noted that the data processing method proposed in the present embodiment can be applied to any application scenario corresponding to any data storage mode, such as object storage application scenarios, block storage application scenarios, and file system storage application scenarios, and the present embodiment does not specifically limit this.

[0025] As Figure 1 shown, the flow can include the following steps:

[0026] Step 101, when detecting the creation instruction of using at least one data storage block on the node as the data storage block of any logical storage space to be created, or when determining to start a snapshot creation of any logical storage space on the node, create a new object mapping table for the logical storage space, and store the new object mapping table locally on the node.

[0027] In the present embodiment, as previously described, the logical storage space refers to the storage space divided based on the available storage space provided by the distributed storage system. For example, in the object storage application scenario, the logical storage space can be considered as a storage bucket; in the block storage application scenario, the logical storage space can be considered as a storage volume; in the file system storage application scenario, the logical storage space can be considered as a file; and so on.

[0028] Among them, the above-mentioned storage volume can refer to the RBD volume in the distributed storage system (i.e. Ceph), which is not specifically limited here; RBD is the abbreviation of RADOS Block Device, and RADOS is the abbreviation of Reliable, Autonomic Distributed Object Store; in Ceph, RBD provides a block storage service interface as a RADOS block device to provide block storage services. As for the above-mentioned storage bucket, file, etc., the present embodiment does not specifically limit this.

[0029] Any object mapping table (also denoted as objectmap) corresponding to any logical storage space stored locally at the node is used to indicate the block usage state of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table.

[0030] The specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table. For example, as an embodiment, the specified time period corresponding to any object mapping table can be the time period between the creation time of the object mapping table and the time when the snapshot creation of the logical storage space is determined to be started; the embodiment is not specifically limited.

[0031] For example, it is assumed that a new object mapping table such as object mapping table A is created for the logical storage space and stored locally at the node when the creation instruction of using at least one data storage block on the node as the data storage block of any logical storage space to be created is detected; then, a new object mapping table such as object mapping table B is created for the logical storage space again when the snapshot creation of any logical storage space on the node is determined to be started, and the object mapping table B is also stored locally at the node; based on this, the specified time period corresponding to the object mapping table A can be the time period between the creation time of the object mapping table A and the time when the snapshot creation of the logical storage space is determined to be started this time. Similarly, when the snapshot creation of the logical storage space on the node is determined to be started again, the specified time period corresponding to the object mapping table B can be the time period between the creation time of the object mapping table B and the time when the snapshot creation of the logical storage space is determined to be started this time.

[0032] In the embodiment, as an embodiment, the block usage state of any data storage block in any object mapping table may, for example, include but is not limited to the first state used to indicate that there is data in the data storage block and the data storage block has undergone a data update operation after the object mapping table is created, the second state used to indicate that there is data in the data storage block and the data storage block has not undergone a data update operation after the object mapping table is created, the third state used to indicate that the data in the data storage block is to be deleted, and the fourth state used to indicate that there is no data in the data storage block, etc.

[0033] In the embodiment, as an embodiment, the above-mentioned creation of a new object mapping table for the logical storage space may, for example, be: first checking whether the object mapping table corresponding to the logical storage space exists in the node locally;

[0034] Afterwards, if the object mapping table corresponding to the logical storage space exists in the local node, the object mapping table corresponding to the logical storage space is obtained, the object mapping table created at the latest time is taken as a reference mapping table, the reference mapping table is copied, the block usage state of the data storage block in the copied object mapping table in the first state is modified to the second state, and the new object mapping table is obtained.

[0035] If the object mapping table corresponding to the logical storage space does not exist in the local node, an object mapping table is created, the object mapping table includes all data storage blocks belonging to the logical storage space on the local node, the block usage state of each data storage block in the object mapping table is set to the fourth state, and the new object mapping table is obtained.

[0036] As to how to detect the creation instruction of using at least one data storage block on the local node as the data storage block of any logical storage space to be created and how to determine to start the snapshot creation of any logical storage space on the local node, examples will be described below, and details are not described herein.

[0037] In step 102, when detecting that any data update operation occurs on any data storage block on the local node, the block usage state of the data storage block in the first object mapping table stored locally is updated based on the data update operation. The first object mapping table refers to the object mapping table created at the latest time corresponding to the logical storage space to which the data storage block belongs.

[0038] In the embodiment, as described above, the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table, which can be the time period between the creation time of the object mapping table and the time when it is determined to start the snapshot creation of the logical storage space. Therefore, when detecting that any data update operation occurs on any data storage block on the local node, the object mapping table to be updated can be the object mapping table created at the latest time among all object mapping tables corresponding to the logical storage space to which the data storage block belongs, that is, the first object mapping table. Based on this, the block usage state of the data storage block in the first object mapping table stored locally is updated based on the data update operation.

[0039] In this embodiment, as an example, when the node receives any data update request, the node can determine at least one data storage block on which the data update operation is to be performed based on the data to be updated indicated by the data update request according to the specified data update manner; here, the data update manner adopted is not specifically limited, and can be flexibly set based on actual application requirements. Based on this, detecting any data update operation on any data storage block on the node in this step may, for example, be that: for any data storage block on the node, when the data storage block is contained in the at least one data storage block on which the data update operation is to be performed which is determined based on the received data update request, it is determined that the data update operation on the data storage block is detected.

[0040] In this embodiment, the data update request refers to a request for indicating data update, which may, for example, include a request for indicating data writing, a request for indicating data deletion, and the like; the data update request corresponds to the data update operation, based on which the data update operation may, for example, include an operation for indicating data writing, an operation for indicating data deletion, and the like; the embodiment is not specifically limited.

[0041] In this embodiment, as an example, the block usage state of the data storage block in the locally stored first object mapping table is updated based on the data update operation in this step, which may, for example, be that: if the data update operation indicates data writing, the block usage state of the data storage block in the first object mapping table is updated to the first state; if the data update operation indicates data deletion, the block usage state of the data storage block in the first object mapping table is updated to the third state; and the like, and the like.

[0042] At this point, the process shown in Figure 1 is completed.

[0043] Through the process shown in Figure 1As shown in the flow, in the embodiments of the present application, when a creation instruction of using at least one data storage block on the node as a data storage block of any logical storage space to be created is detected, or when it is determined to start a snapshot creation on any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node. Based on this, when any data update operation occurs on any data storage block on the node, frequent interaction with other nodes such as a metadata node is not needed, but the block usage state of the data storage block in the locally stored first object mapping table can be directly updated on the node based on the data update operation, so that no additional cross-node interaction is needed, which not only improves the efficiency of data processing, but also ensures the consistency of data update and block usage state update of the data storage block, and improves the performance of data processing.

[0044] In addition, in the embodiments of the present application, the object mapping table used to indicate the block usage state of the data storage block on any storage node is stored locally on the storage node, and does not need to be synchronized to other nodes such as a metadata node for storage, so that when data loss is caused by a failure of the metadata node or one of the storage nodes, the data such as the object mapping table on other storage nodes will not be affected by the failed node, thereby reducing the data pollution range.

[0045] The above step 101 is further described as follows:

[0046] In the embodiments, as an example, the detection of the creation instruction of using at least one data storage block on the node as a data storage block of any logical storage space to be created may be, for example, that when a logical storage space creation request indicating that at least one data storage block on the node is used as a data storage block of any logical storage space to be created is received, it is determined that the creation instruction is detected.

[0047] In the embodiments, as an example, the determination to start a snapshot creation on any logical storage space on the node may be, for example, that when a snapshot creation request indicating a snapshot creation on any logical storage space on the node is received, it is determined to start a snapshot creation on the logical storage space; or, for any logical storage space on the node, the node triggers a snapshot creation on the logical storage space periodically, and based on this, it is determined to start a snapshot creation on the logical storage space each time the node triggers a snapshot creation on the logical storage space; and the like.

[0048] In the embodiments, as an example, referring to Figure 2 As shown, when it is determined to start a snapshot creation on any logical storage space on the node, the method further includes the following steps:

[0049] Step 201, controlling the node to suspend receiving data update requests.

[0050] Step 202, when there still exists at least one received but not yet processed data update request for the logical storage space indicated by the data update request, for each unprocessed data update request, updating the locally stored second object mapping table based on the unprocessed data update request, and performing persistent processing on the data to be updated indicated by the unprocessed data update request and the second object mapping table according to a specified persistent mode.

[0051] In the embodiment, the second object mapping table refers to the object mapping table corresponding to the logical storage space in the local node, which is the latest in creation time.

[0052] As to how to update the locally stored second object mapping table based on the unprocessed data update request, please refer to the related description in step 102 above, which will not be repeated here. As to how to perform persistent processing on the data to be updated indicated by the unprocessed data update request and the second object mapping table according to a specified persistent mode, examples will be described below, which will not be repeated here.

[0053] Step 203, after the data indicated by each unprocessed data update request and the second object mapping table are all processed, creating a snapshot of the logical storage space.

[0054] In the embodiment, as an example, for any logical storage space on the node, the object mapping table corresponding to the logical storage space in the local node can be stored in a key-value (KV) storage mode, where K can include the space identifier of the logical storage space (such as the volume identifier of a storage volume, the bucket identifier of a storage bucket, etc.), and the snapshot identifier of the snapshot of the logical storage space, etc., and V can include the object mapping table corresponding to the logical storage space, which is not specifically limited in the embodiment. Based on this, as an example, after creating the snapshot of the logical storage space, the second object mapping table can be used as the object mapping table corresponding to the snapshot of the logical storage space, and at this time, the K in the KV indicating the second object mapping table in the local node can be updated from the space identifier of the logical storage space to the space identifier of the logical storage space and the snapshot identifier of the snapshot of the logical storage space.

[0055] In the embodiment, after the second object mapping table is used as the object mapping table corresponding to the snapshot of the logical storage space, a new object mapping table is created for the logical storage space.

[0056] And, after storing the new object mapping table locally, a first message indicating that the current snapshot creation of the logical storage space has been completed is sent to the metadata node; if a second message indicating that the snapshot creation is successful is received from the metadata node within a set time period, it is determined that the current snapshot creation of the logical storage space is successful, and the node continues to receive data update requests after it is determined that the current snapshot creation of the logical storage space is successful; otherwise, it is determined that the current snapshot creation of the logical storage space fails, and the new object mapping table is deleted from the local node, and the K in the KV indicating the second object mapping table in the local node is updated from the space identifier of the logical storage space and the snapshot identifier of the current snapshot of the logical storage space to the space identifier of the logical storage space. The second message is sent by the metadata node after receiving the first message from all storage nodes where the logical storage space is located.

[0057] In the embodiment, the object mapping table indicating the block usage state of the data storage block on any storage node is stored locally on the storage node, so that the persistence of the corresponding data can be implemented locally on the node, without waiting for the object mapping tables of all storage nodes to be synchronized to other nodes such as the metadata node and then uniformly subjected to the persistence processing, thereby reducing the frequent interaction between nodes and effectively improving the efficiency of data processing.

[0058] The step 102 is further described as follows:

[0059] In the embodiment, as an example, in order to ensure the recoverability of the data, after updating the block usage state of the data storage block in the locally stored first object mapping table based on the data update operation, the data to be updated indicated by the data update operation and the first object mapping table are subjected to the persistence processing according to a specified persistence mode, for subsequent data recovery. Specifically, the data to be updated indicated by the data update operation and the first object mapping table can be synchronously written into a specified persistent storage medium to implement the persistence of the data; the embodiment is not specifically limited.

[0060] In the embodiment, the object mapping table indicating the block usage state of the data storage block on any storage node is stored locally on the storage node, so that the persistence of the corresponding data can be implemented locally on the node, without waiting for the object mapping tables of all storage nodes to be synchronized to other nodes such as the metadata node and then uniformly subjected to the persistence processing, thereby reducing the frequent interaction between nodes and effectively improving the efficiency of data processing.

[0061] The following describes how to perform the corresponding data processing when it is determined to start the incremental replication of the data in any logical storage space to be replicated on the node:

[0062] In this embodiment, as an example, refer to Figure 3 As shown in the figure, when determining to start the incremental replication of the data in any to-be-replicated logical storage space on the node, the method can further include the following steps:

[0063] Step 301, creating a snapshot of the to-be-replicated logical storage space.

[0064] Step 302, after determining that the snapshot creation of the to-be-replicated logical storage space is successful, obtaining a third object mapping table and a fourth object mapping table corresponding to the to-be-replicated logical storage space from the local.

[0065] The third object mapping table refers to the object mapping table corresponding to the latest snapshot of the to-be-replicated logical storage space. Here, the latest snapshot of the to-be-replicated logical storage space can be understood as the snapshot with the latest snapshot creation time among the snapshots of the to-be-replicated logical storage space.

[0066] The fourth object mapping table refers to the object mapping table corresponding to the snapshot created at the last data replication of the to-be-replicated logical storage space.

[0067] In this embodiment, as described above, the object mapping table corresponding to the logical storage space is stored in the local in the storage mode of KV, where K can include the space identifier of the logical storage space and the snapshot identifier of the snapshot of the logical storage space, etc. Based on this, the object mapping table corresponding to the latest snapshot of the to-be-replicated logical storage space can be obtained from the local as the third object mapping table, and the object mapping table corresponding to the snapshot created at the last data replication of the to-be-replicated logical storage space can be obtained from the local as the fourth object mapping table.

[0068] As to how to determine whether the snapshot creation of the to-be-replicated logical storage space is successful in this step, refer to the related description above, which will not be repeated here.

[0069] Step 303, for each data storage block belonging to the to-be-replicated logical storage space on the node, checking whether the block usage state of the data storage block in the third object mapping table is consistent with the block usage state of the data storage block in the fourth object mapping table, if not, determining that the data storage block has undergone a data update operation since the last data replication of the to-be-replicated logical storage space, and determining the data storage block as a to-be-replicated data storage block; and after checking all the data storage blocks belonging to the to-be-replicated logical storage space on the node, performing incremental replication on the data in the to-be-replicated logical storage space according to the at least one to-be-replicated data storage block determined.

[0070] In the embodiment, the incremental replication refers to replicating only the data changed since the last data replication in the data replication process, such as the data added, modified or deleted since the last data replication. Therefore, the embodiment can obtain the data storage block changed since the last data replication of the logical storage space to be replicated on the node, i.e., the data storage block to be replicated, by comparing the third object mapping table and the fourth object mapping table; based on this, the data in the logical storage space to be replicated can be incrementally replicated according to the at least one data storage block to be replicated determined.

[0071] For example, as an embodiment, the data in the logical storage space to be replicated can be incrementally replicated according to the at least one data storage block to be replicated determined, which can be implemented as follows: the node can maintain a difference bitmap corresponding to the logical storage space to be replicated in the local, the difference bitmap being used to indicate whether each data storage block belonging to the logical storage space to be replicated on the node has changed since the last data replication of the logical storage space to be replicated, and the difference bitmap can be determined based on the at least one data storage block to be replicated determined; based on this, when the data in the logical storage space to be replicated is incrementally replicated, the difference bitmap corresponding to the logical storage space to be replicated can be obtained first, so that the data in the logical storage space to be replicated is incrementally replicated based on the difference bitmap.

[0072] In the embodiment, the object mapping table used to indicate the block usage state of the data storage block on any storage node is stored locally on the storage node, so that when the incremental replication of any logical storage space is performed, each storage node where the logical storage space is located can independently determine the data changed since the last data replication of the logical storage space, so as to perform the incremental replication of the logical storage space, without additional cross-node interaction, thereby effectively improving the efficiency of data replication.

[0073] The following describes how to query the used storage space capacity of any logical storage space to be queried and output by using the object mapping table when the third message indicating the query of the used storage space capacity of any logical storage space to be queried is received:

[0074] In the embodiment, as an embodiment, when the third message indicating the query of the used storage space capacity of any logical storage space to be queried is received, the object mapping table corresponding to the latest snapshot of the logical storage space to be queried is obtained from the local of the node first;

[0075] Afterwards, a sum value of the number of data storage blocks in the first state and the number of data storage blocks in the second state is determined based on the obtained object mapping table, so as to determine the used storage space capacity in the storage space corresponding to each data storage block belonging to the logical storage space to be queried based on the sum value and the size of the data storage block of the logical storage space to be queried, and send a fourth message indicating the space capacity to the metadata node; here, the size of each data storage block in any logical storage space is the same.

[0076] Wherein, after receiving the fourth messages sent by all the storage nodes where the logical storage space to be queried is located, the metadata node determines the used storage space capacity of the logical storage space to be queried according to the sum value of the space capacities indicated by the fourth messages and outputs.

[0077] The object mapping table is further described as follows:

[0078] In this embodiment, as an example, all the data storage blocks of any logical storage space are grouped into multiple disposition groups. Based on this, the above-mentioned creation of a new object mapping table for the logical storage space can specifically include: creating a new object mapping table for each disposition group of the logical storage space respectively; wherein any object mapping table corresponding to any disposition group is used to indicate the block usage state of at least one data storage block belonging to the disposition group on the node within the specified time period corresponding to the object mapping table.

[0079] In this embodiment, based on the above description, it can be known that the object mapping table corresponding to the logical storage space is stored in the local according to the storage mode of KV, wherein K can include the space identifier of the logical storage space and the snapshot identifier of the snapshot of the logical storage space; based on this, K can also include the group identifier of the disposition group.

[0080] For any disposition group of any logical storage space on the node, the specified operation of any object mapping table corresponding to the disposition group is independent of the specified operation of any object mapping table corresponding to other disposition groups; wherein the specified operation at least includes access, update, and persistence, etc., which are not specifically limited in this embodiment.

[0081] In this embodiment, by further refining K in KV to the disposition group level, when performing data access, data update, and data persistence operations of the logical storage space, the specified operations (such as access, update, or persistence, etc.) of the object mapping tables of each disposition group corresponding to the logical storage space can be executed concurrently, that is, independently and without affecting each other, which can further improve the efficiency of data processing.

[0082] This concludes the description of the method provided in the embodiments of this application. The apparatus provided in the embodiments of this application will now be described:

[0083] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 4 As shown, the data processing device 400 includes a creation module 401 and an update module 402;

[0084] The creation module 401 is used to create a new object mapping table for any logical storage space when it detects a creation instruction that uses at least one data storage block on the local node as a data storage block for any logical storage space to be created, or when it is determined to initiate a snapshot creation for any logical storage space on the local node, and to store the new object mapping table locally on the local node; any object mapping table corresponding to any logical storage space already stored locally on the local node is used to indicate the block usage status of at least one data storage block on the local node belonging to the logical storage space within a specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined based on the creation time of the object mapping table;

[0085] The update module 402 is used to update the block usage status of any data storage block in the locally stored first object mapping table when any data update operation is detected in any data storage block on the local node. The first object mapping table refers to the object mapping table in the object mapping table corresponding to the logical storage space to which the data storage block belongs, which is the object mapping table with the latest creation time.

[0086] As an example, after updating the block usage status of the data storage block in the locally stored first object mapping table based on the data update operation, the apparatus further includes: a persistence module, used to persist the data to be updated indicated by the data update operation and the first object mapping table according to a specified persistence method.

[0087] As an example, the creation module 401 is further configured to: when determining to initiate the creation of a snapshot of any logical storage space on the local node, control the local node to suspend receiving data update requests;

[0088] when there is at least one received but not yet processed data update request and the indicated data to be updated is the data update request of the logical storage space, for each data update request, updating the stored second object mapping table based on the data update request, and persisting the data to be updated indicated by the data update request and the second object mapping table according to the specified persistence mode; the second object mapping table refers to the object mapping table corresponding to the logical storage space with the latest creation time;

[0089] after the data indicated by each data update request and the second object mapping table are all persisted, creating the current snapshot of the logical storage space.

[0090] As an embodiment, the creating module 401 is further configured to: after the current snapshot of the logical storage space is created, record the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space.

[0091] After the second object mapping table is recorded as the object mapping table corresponding to the current snapshot of the logical storage space, a new object mapping table is created for the logical storage space, and after the new object mapping table is stored locally, a first message indicating that the current snapshot of the logical storage space has been created is sent to the metadata node, if a second message indicating that the snapshot creation is successful is received from the metadata node within a set time period, it is determined that the current snapshot of the logical storage space is created successfully, otherwise, it is determined that the current snapshot of the logical storage space is created unsuccessfully, and the new object mapping table is deleted from the local node; the second message is sent by the metadata node after receiving the first message sent by all storage nodes where the logical storage space is located.

[0092] As an embodiment, the apparatus further comprises a replicating module configured to: when it is determined to start the incremental replication of the data in any logical storage space to be replicated on the local node, create a snapshot of the logical storage space to be replicated.

[0093] After it is determined that the snapshot of the logical storage space to be replicated is created successfully, a third object mapping table and a fourth object mapping table corresponding to the logical storage space to be replicated are obtained from the local node; the third object mapping table refers to the object mapping table corresponding to the latest snapshot of the logical storage space to be replicated; the fourth object mapping table refers to the object mapping table corresponding to the snapshot created during the last data replication of the logical storage space to be replicated.

[0094] For each data storage block belonging to the logical storage space to be replicated on the node, it is checked whether the block usage state of the data storage block in the third object mapping table is consistent with the block usage state of the data storage block in the fourth object mapping table, if not, it is determined that the data storage block has undergone a data update operation since the last data replication of the logical storage space to be replicated, and the data storage block is determined as a data storage block to be replicated; and after checking each data storage block belonging to the logical storage space to be replicated on the node, the data in the logical storage space to be replicated is incrementally replicated according to the at least one data storage block to be replicated determined.

[0095] As an embodiment, all data storage blocks of any logical storage space are grouped into multiple homing groups;

[0096] Creating a new object mapping table for the logical storage space includes: creating a new object mapping table for each homing group of the logical storage space respectively; wherein any object mapping table corresponding to any homing group is used to indicate the block usage state of at least one data storage block belonging to the homing group on the node within a specified time period corresponding to the object mapping table; for any homing group of any logical storage space on the node, the specified operation of any object mapping table corresponding to the homing group is independent of the specified operation of any object mapping table corresponding to other homing groups; the specified operation at least includes access, update, and persistence.

[0097] At this point, the process of Figure 4 The structure of the device is described.

[0098] The implementation process of the functions and roles of each module in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0099] For the device embodiment, since it basically corresponds to the method embodiment, please refer to the part of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components can be or can not be physically separated, and the components displayed as modules can be or can not be physical modules, that is, they can be located in one place, or they can be distributed on multiple network modules. According to actual needs, part or all of the modules can be selected to achieve the purpose of the scheme of the present application. Those skilled in the art can understand and implement without creative labor.

[0100] Please refer to Figure 5A hardware structure schematic diagram of an electronic device is provided for an exemplary embodiment of the present application. The electronic device can include a processor 501, a communication interface 502, a computer readable storage medium 503 and a communication bus 504. The processor 501, the communication interface 502 and the computer readable storage medium 503 complete the communication among each other through the communication bus 504. Among them, the computer readable storage medium 503 stores computer program instructions; the processor 501 can execute the steps of the method described in the above embodiment by executing the computer program instructions stored in the computer readable storage medium 503. The electronic device can also include other hardware according to the actual function of the electronic device, which will not be described here.

[0101] Correspondingly, the present application also provides a computer readable storage medium, which stores a plurality of computer program instructions, and the computer program instructions can implement the method disclosed in the above exemplary embodiments of the present application when executed by a processor.

[0102] Exemplarily, the above computer readable storage medium can be any electronic, magnetic, optical or other physical storage apparatus, and can contain or store information such as executable instructions, data, etc. For example, the computer readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state disk, any type of storage disk (such as optical disk, dvd, etc.), or similar storage medium, or combination thereof. The processor and the memory can be supplemented by or incorporated into a special logic circuit.

[0103] The above is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized by, The method is applied to any storage node in a distributed storage system, and comprises: When a creation instruction of using at least one data storage block on the node as a data storage block of any logical storage space to be created is detected, or when it is determined to start creating a snapshot of any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate a block usage state of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined according to a creation time of the object mapping table; When any data update operation is detected on any data storage block on the node, the block usage state of the data storage block in a first object mapping table stored locally is updated based on the data update operation; the first object mapping table refers to an object mapping table corresponding to a logical storage space to which the data storage block belongs and having a latest creation time.

2. The method of claim 1, wherein, After the block usage state of the data storage block in the first object mapping table stored locally is updated based on the data update operation, the method further comprises: The data to be updated indicated by the data update operation and the first object mapping table are subjected to persistent processing in a specified persistent manner.

3. The method of claim 1, wherein, When it is determined to start creating a snapshot of any logical storage space on the node, the method further comprises: The node is controlled to suspend receiving data update requests; When there is at least one received but not yet processed data update request and the logical storage space indicated by the data update request is the logical storage space, for each unprocessed data update request, a second object mapping table stored locally is updated based on the unprocessed data update request, and the data to be updated indicated by the unprocessed data update request and the second object mapping table are subjected to persistent processing in a specified persistent manner; the second object mapping table refers to an object mapping table corresponding to the logical storage space and having a latest creation time; After the data indicated by each unprocessed data update request and the second object mapping table are subjected to persistent processing, a current snapshot of the logical storage space is created.

4. The method of claim 3, further comprising: After the current snapshot of the logical storage space is created, the second object mapping table is used as an object mapping table corresponding to the current snapshot of the logical storage space. The second object mapping table is stored in the local node, and a first message indicating that the current snapshot of the logical storage space has been created is sent to the metadata node. If a second message indicating that the snapshot creation is successful is received from the metadata node within a set time period, it is determined that the current snapshot of the logical storage space is successfully created, otherwise, it is determined that the current snapshot of the logical storage space is unsuccessfully created, and the new object mapping table is deleted from the local node. The second message is sent by the metadata node after receiving the first message from all storage nodes where the logical storage space is located.

5. The method of claim 4, wherein, The method further comprises: When it is determined to start the incremental replication of data in any to-be-replicated logical storage space on the local node, a snapshot of the to-be-replicated logical storage space is created; After it is determined that the snapshot of the to-be-replicated logical storage space is successfully created, a third object mapping table and a fourth object mapping table corresponding to the to-be-replicated logical storage space are obtained from the local node. The third object mapping table corresponds to a latest snapshot of the to-be-replicated logical storage space, and the fourth object mapping table corresponds to a snapshot created during the last data replication of the to-be-replicated logical storage space; For each data storage block belonging to the to-be-replicated logical storage space on the local node, it is checked whether the block usage state of the data storage block in the third object mapping table is consistent with the block usage state of the data storage block in the fourth object mapping table. If not, it is determined that the data storage block has undergone a data update operation since the last data replication of the to-be-replicated logical storage space, and the data storage block is determined to be a to-be-replicated data storage block. After checking all data storage blocks belonging to the to-be-replicated logical storage space on the local node, the data in the to-be-replicated logical storage space is incrementally replicated according to the at least one to-be-replicated data storage block.

6. The method of claim 1, wherein, All data storage blocks of any logical storage space are grouped into multiple home groups. The new object mapping table for the logical storage space comprises: a new object mapping table is created for each home group of the logical storage space. Any object mapping table corresponding to any home group is used to indicate the block usage state of at least one data storage block belonging to the home group on the local node within a specified time period corresponding to the object mapping table. For any home group of any logical storage space on the local node, the specified operation of any object mapping table corresponding to the home group is independent of the specified operation of any object mapping table corresponding to other home groups. The specified operation at least includes access, update, and persistence.

7. A data processing apparatus, characterized by The device is applied to any storage node in a distributed storage system, and the device comprises: The creating module is configured to create a new object mapping table for any logical storage space on the node and store the new object mapping table locally on the node when detecting a creation instruction of using at least one data storage block on the node as a data storage block of any logical storage space to be created, or when determining to start a snapshot creation of any logical storage space on the node. Any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate a block usage state of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table. The specified time period corresponding to any object mapping table is determined according to a creation time of the object mapping table. The updating module is configured to update a block usage state of any data storage block in a first object mapping table stored locally based on any data update operation of the data storage block when detecting the data update operation. The first object mapping table refers to an object mapping table corresponding to a logical storage space to which the data storage block belongs and having a latest creation time.

8. The apparatus of claim 7, wherein, After updating the block usage state of the data storage block in the first object mapping table stored locally based on the data update operation, the apparatus further comprises a persisting module configured to perform a persisting process on data to be updated indicated by the data update operation and the first object mapping table according to a specified persisting manner; and / or The creating module is further configured to control the node to suspend receiving data update requests when determining to start a snapshot creation of any logical storage space on the node, update a second object mapping table stored locally based on each unprocessed data update request when there is at least one unprocessed data update request received but not processed and a logical storage space indicated by the unprocessed data update request is the logical storage space, perform a persisting process on data to be updated indicated by the unprocessed data update request and the second object mapping table according to a specified persisting manner, create a snapshot of the logical storage space when the data indicated by each unprocessed data update request and the second object mapping table are all processed, and / or The creating module is further configured to: after creating the current snapshot of the logical storage space, take the second object mapping table as an object mapping table corresponding to the current snapshot of the logical storage space; wherein, after taking the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space, a new object mapping table is created for the logical storage space; and after storing the new object mapping table locally, a first message indicating that the current snapshot of the logical storage space has been created is sent to the metadata node; if a second message indicating that the snapshot creation is successful is received from the metadata node within a set time period, it is determined that the current snapshot of the logical storage space is created successfully, otherwise, it is determined that the current snapshot of the logical storage space is created unsuccessfully, and the new object mapping table is deleted from the local; the second message is sent by the metadata node after receiving the first message from all storage nodes where the logical storage space is located; and / or, The apparatus further comprises a copying module configured to: when it is determined to start the incremental replication of data in any to-be-replicated logical storage space on the node, create a snapshot of the to-be-replicated logical storage space; after it is determined that the snapshot of the to-be-replicated logical storage space is created successfully, obtain a third object mapping table and a fourth object mapping table corresponding to the to-be-replicated logical storage space from the local; wherein, the third object mapping table is an object mapping table corresponding to the latest snapshot of the to-be-replicated logical storage space; the fourth object mapping table is an object mapping table corresponding to a snapshot created at the last data replication of the to-be-replicated logical storage space; for each data storage block belonging to the to-be-replicated logical storage space on the node, check whether the block usage state of the data storage block in the third object mapping table is consistent with the block usage state of the data storage block in the fourth object mapping table, if not, determine that the data storage block has undergone a data update operation since the last data replication of the to-be-replicated logical storage space, and determine the data storage block as a to-be-replicated data storage block; and after checking all data storage blocks belonging to the to-be-replicated logical storage space on the node, perform the incremental replication of data in the to-be-replicated logical storage space according to the at least one to-be-replicated data storage block determined.

9. The apparatus of claim 7, wherein, All data storage blocks of any logical storage space are grouped into multiple homing groups; The creating of the new object mapping table for the logical storage space comprises: creating a new object mapping table for each homing group of the logical storage space respectively; wherein, any object mapping table corresponding to any homing group is used to indicate the block usage state of at least one data storage block belonging to the homing group on the node within a specified time period corresponding to the object mapping table; for any homing group of any logical storage space on the node, the specified operation of any object mapping table corresponding to the homing group is independent of the specified operation of any object mapping table corresponding to other homing groups; the specified operation at least comprises access, update, and persistence.

10. An electronic device, comprising: The electronic device comprises: a processor; and A computer readable storage medium having stored therein computer program instructions, which, when executed by the processor, cause the processor to perform the steps of any of the methods of claims 1 to 6.

Images