Data processing method and device and electronic equipment

Abstract

The embodiment of the invention provides a data processing method and device and electronic equipment. In the embodiment of the invention, when a creation instruction of any logic storage space is detected, or when it is determined that snapshot creation of any logic storage space on the node is started, a new object mapping table is created for the logic storage space, and the new object mapping table is stored to the local of the node; on the basis, when any data updating operation of any data storage block on the node is detected, frequent interaction with other nodes such as a metadata node is not needed, and the block use state of the data storage block in the locally stored first object mapping table is directly updated on the node on the basis of the data updating operation; in this way, extra cross-node interaction is not needed, the data processing efficiency is improved, the consistency of data updating and block use state updating of the data storage block is guaranteed, and the data processing performance is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and electronic device. Background Art

[0002] In a distributed storage system, data can be stored in object storage, block storage, or file system storage. The available storage space provided by the distributed storage system is divided into at least one logical storage space, and each logical storage space is divided into multiple data storage blocks for data storage; the data storage blocks of each logical storage space may be distributed on multiple different storage nodes.

[0003] In actual applications of data processing using any logical storage space, the block usage status of the data storage blocks of the logical storage space (such as the status of no data, data, and data to be deleted in the data storage block) needs to be recorded as metadata information of the logical storage space to the metadata node; based on this, while updating the data in the logical storage space, the block usage status of the data storage blocks of the logical storage space on the metadata node must also be updated as synchronously as possible to ensure data consistency. However, since the data storage blocks of a logical storage space may be distributed on multiple storage nodes, multiple storage nodes may be involved when updating data (such as data writing, etc.) on the logical storage space. In this way, when the storage nodes involved perform the corresponding data update operations concurrently, they can only interact with the metadata nodes in turn to realize the update of the block usage status of the data storage blocks of the logical storage space, and the consistency of the data update and the block usage status update of the data storage block cannot be guaranteed, thereby affecting the performance of data processing. Summary of the invention

[0004] In view of this, the present application provides a data processing method, device and electronic device to ensure the consistency of data update of logical storage space and block usage status update of data storage block, so as to improve the performance of data processing.

[0005] The present application provides a data processing method, which is applied to any storage node in a distributed storage system; the method includes:

[0006] When a creation indication of using at least one data storage block on the node as a data storage block of any logical storage space to be created is detected, or when it is determined to start creating a snapshot of any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate the block usage status of at least one data storage block belonging to the logical storage space on the node within the specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table;

[0007] When any data update operation is detected in any data storage block on this node, the block usage status of the data storage block in the locally stored first object mapping table is updated based on the data update operation; the first object mapping table refers to the object mapping table with the latest creation time in the object mapping table corresponding to the logical storage space to which the data storage block belongs.

[0008] The embodiment of the present application further provides a data processing device, which is applied to any storage node in a distributed storage system; the device includes:

[0009] A creation module, used for, when detecting a creation indication of using at least one data storage block on the node as a data storage block of any logical storage space to be created, or when determining to start a snapshot creation of any logical storage space on the node, creating a new object mapping table for the logical storage space and storing the new object mapping table locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate the block usage status of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table;

[0010] An update module is used to update the block usage status of the data storage block in the first object mapping table stored locally based on the data update operation when any data update operation is detected to occur in any data storage block on this node; the first object mapping table refers to the object mapping table with the latest creation time in the object mapping table corresponding to the logical storage space to which the data storage block belongs.

[0011] The present application also provides an electronic device, the electronic device comprising:

[0012] Processor; and

[0013] A computer-readable storage medium stores computer program instructions, which, when executed by a processor, cause the processor to execute the steps of the above method.

[0014] An embodiment of the present application further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps in the above method.

[0015] It can be seen from the above technical solution that in an embodiment of the present application, when a creation indication is detected for using at least one data storage block on this node as a data storage block for any logical storage space to be created, or when it is determined to start a snapshot creation of any logical storage space on this node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on this node; based on this, when any data update operation is detected for any data storage block on this node, there is no need to frequently interact with other nodes such as metadata nodes, but the block usage status of the data storage block in the first object mapping table stored locally can be updated directly on this node based on the data update operation. This does not require additional cross-node interaction, which not only improves the efficiency of data processing, but also ensures the consistency of data updates and block usage status updates of data storage blocks, thereby improving data processing performance.

[0016] Furthermore, in an embodiment of the present application, an object mapping table for indicating the block usage status of a data storage block on any storage node is stored locally on the storage node and does not need to be synchronized to other nodes such as a metadata node for storage. In this way, when a metadata node or one of the storage nodes fails and causes data loss, data on other storage nodes such as the object mapping table will not be affected by the failed node, thereby reducing the scope of data pollution. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0018] Figure 1 A flow chart of the method provided in an embodiment of the present application.

[0019] Figure 2 Another method flow chart provided for an embodiment of the present application.

[0020] Figure 3 Another method flow chart provided for an embodiment of the present application.

[0021] Figure 4 A schematic diagram of the device structure provided in an embodiment of the present application.

[0022] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0024] See also Figure 1 , Figure 1 A flow chart of the method provided in an embodiment of the present application. The method is applied to any storage node in a distributed storage system. In this embodiment, the storage node may be, for example, an electronic device such as a server, which is not specifically limited in this embodiment. It should be noted that the data processing method proposed in this embodiment can be applied to any application scenario corresponding to any data storage method, such as object storage application scenarios, block storage application scenarios, and file system storage application scenarios, which are not specifically limited in this embodiment.

[0025] like Figure 1 As shown, the process may include the following steps:

[0026] Step 101, when a creation indication is detected to use at least one data storage block on this node as a data storage block for any logical storage space to be created, or when it is determined to start a snapshot creation of any logical storage space on this node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on this node.

[0027] In this embodiment, as described above, the logical storage space refers to the storage space divided based on the available storage space provided by the distributed storage system. For example, in the object storage application scenario, the logical storage space can be considered to be a storage bucket; in the block storage application scenario, the logical storage space can be considered to be a storage volume; in the file system storage application scenario, the logical storage space can be considered to be a file; and so on.

[0028] The storage volume may refer to an RBD volume in a distributed storage system (i.e., Ceph), which is not specifically limited here; RBD is the abbreviation of RADOS Block Device, and RADOS is the abbreviation of Reliable, Autonomic Distributed Object Store; in Ceph, RBD, as a RADOS block device, provides a block storage service interface to the outside world for providing block storage services. As for the storage buckets, files, etc., this embodiment does not specifically limit them.

[0029] Any object mapping table (also referred to as objectmap) corresponding to any logical storage space stored locally on this node is used to indicate the block usage status of at least one data storage block belonging to the logical storage space on this node within a specified time period corresponding to the object mapping table.

[0030] The specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table. For example, as an embodiment, the specified time period corresponding to any object mapping table can be the time period between the creation time of the object mapping table and the time when the snapshot creation of the logical storage space is determined to be started once; this embodiment does not specifically limit this.

[0031] Exemplarily, it is assumed that when a creation instruction is detected to use at least one data storage block on the node as a data storage block of any logical storage space to be created, a new object mapping table such as object mapping table A is created for the logical storage space, and the object mapping table A is stored locally on the node; thereafter, when it is determined to start the creation of a snapshot of any logical storage space on the node, a new object mapping table such as object mapping table B is created for the logical storage space, and the object mapping table B is also stored locally on the node; based on this, the specified time period corresponding to the object mapping table A can be the time period between the creation time of the object mapping table A and the time when the creation of the snapshot of the logical storage space is determined to be started this time. And by analogy, thereafter, when it is determined to start the creation of a snapshot of the logical storage space on the node again, the specified time period corresponding to the object mapping table B can be the time period between the creation time of the object mapping table B and the time when the creation of the snapshot of the logical storage space is determined to be started this time.

[0032] In this embodiment, as an embodiment, the block usage status of any data storage block in any object mapping table may include, but is not limited to: a first state used to indicate that there is data in the data storage block and a data update operation has occurred in the data storage block after the object mapping table is created, a second state used to indicate that there is data in the data storage block and no data update operation has occurred in the data storage block after the object mapping table is created, a third state used to indicate that the data in the data storage block is to be deleted, and a fourth state used to indicate that there is no data in the data storage block, and so on.

[0033] In this embodiment, as an embodiment, the above-mentioned creation of a new object mapping table for the logical storage space may be specifically implemented as follows: firstly checking whether there is an object mapping table corresponding to the logical storage space in the local node;

[0034] Afterwards, if the object mapping table corresponding to the logical storage space exists locally in the node, the object mapping table with the latest creation time among the mapping tables corresponding to the logical storage space is obtained as a reference mapping table, and the reference mapping table is copied, and the block usage status of the data storage blocks in the copied object mapping table whose block usage status is the first status is modified to the second status, so as to obtain the new object mapping table;

[0035] If the object mapping table corresponding to the logical storage space does not exist locally in this node, then an object mapping table is created, which includes all data storage blocks belonging to the logical storage space on this node, and the block usage status of each data storage block in the object mapping table is set to the above-mentioned fourth state to obtain the above-mentioned new object mapping table.

[0036] As for how to detect in this step the creation indication of using at least one data storage block on this node as the data storage block of any logical storage space to be created, and how to determine to start the creation of a snapshot of any logical storage space on this node, the following will be described with examples and will not be repeated here.

[0037] Step 102, when any data update operation is detected on any data storage block on this node, the block usage status of the data storage block in the locally stored first object mapping table is updated based on the data update operation; the first object mapping table refers to the object mapping table with the latest creation time in the object mapping table corresponding to the logical storage space to which the data storage block belongs.

[0038] In this embodiment, based on the above description, it can be known that the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table, such as the time period between the creation time of the object mapping table and the time when the snapshot creation of the logical storage space is determined to be started. Therefore, when any data update operation is detected for any data storage block on this node, the object mapping table that needs to be updated accordingly can be the most recently created object mapping table among all object mapping tables corresponding to the logical storage space to which the data storage block belongs that has been stored locally at this node, that is, the first object mapping table mentioned above; based on this, the block usage status of the data storage block in the first object mapping table that has been stored locally is updated based on the data update operation.

[0039] In this embodiment, as an embodiment, when this node receives any data update request, this node can determine at least one data storage block on this node to be updated with data according to the specified data update method based on the data to be updated indicated by the data update request; the data update method adopted is not specifically limited here, and can be flexibly set based on actual application requirements. Based on this, in this step, any data update operation is detected on any data storage block on this node. In the specific implementation, for example, it can be: for any data storage block on this node, when the data storage block is included in at least one data storage block on this node to be updated with data based on the received data update request, it is determined that the data storage block is detected to have a data update operation.

[0040] In this embodiment, the above-mentioned data update request refers to a request for indicating data update, such as a request for indicating data writing and a request for indicating data deletion, etc.; the data update request corresponds to the data update operation, based on this, the above-mentioned data update operation may include, for example, an operation for indicating data writing and an operation for indicating data deletion, etc.; this embodiment does not specifically limit it.

[0041] In this embodiment, as an embodiment, in this step, the block usage status of the data storage block in the locally stored first object mapping table is updated based on the data update operation. In the specific implementation, for example, it can be: if the data update operation indicates data writing, then the block usage status of the data storage block in the first object mapping table is updated to the above-mentioned first state; if the data update operation indicates data deletion, then the block usage status of the data storage block in the first object mapping table is updated to the above-mentioned third state; and so on.

[0042] So far, completed Figure 1 The process shown.

[0043] pass Figure 1It can be seen from the shown process that in an embodiment of the present application, when a creation indication is detected to use at least one data storage block on this node as a data storage block for any logical storage space to be created, or when it is determined to start a snapshot creation of any logical storage space on this node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on this node; based on this, when any data update operation is detected for any data storage block on this node, there is no need to frequently interact with other nodes such as metadata nodes, but the block usage status of the data storage block in the first object mapping table stored locally can be updated directly on this node based on the data update operation. This does not require additional cross-node interactions, which not only improves the efficiency of data processing, but also ensures the consistency of data updates and block usage status updates of data storage blocks, thereby improving data processing performance.

[0044] Furthermore, in an embodiment of the present application, an object mapping table for indicating the block usage status of a data storage block on any storage node is stored locally on the storage node and does not need to be synchronized to other nodes such as a metadata node for storage. In this way, when a metadata node or one of the storage nodes fails and causes data loss, data on other storage nodes such as the object mapping table will not be affected by the failed node, thereby reducing the scope of data pollution.

[0045] The above step 101 is further described below:

[0046] In this embodiment, as an embodiment, the above-mentioned detection of a creation indication of using at least one data storage block on the current node as a data storage block for any logical storage space to be created may be specifically implemented as follows: when a logical storage space creation request indicating using at least one data storage block on the current node as a data storage block for any logical storage space to be created is received, it is determined that the above-mentioned creation indication is detected.

[0047] In this embodiment, as an embodiment, the above-mentioned determination to start creating a snapshot of any logical storage space on the current node may be specifically implemented as follows: when a snapshot creation request indicating the creation of a snapshot of any logical storage space on the current node is received, it is determined to start creating a snapshot of the logical storage space; or, for any logical storage space on the current node, the current node may periodically trigger the creation of a snapshot of the logical storage space, based on which, each time the current node triggers the creation of a snapshot of the logical storage space, it is determined to start creating a snapshot of the logical storage space; and so on.

[0048] In this embodiment, as an example, see Figure 2 As shown, when it is determined to start creating a snapshot of any logical storage space on the node, the method may further include the following steps:

[0049] Step 201, controlling the node to stop receiving data update requests.

[0050] Step 202, when there is currently at least one unprocessed data update request that has been received but not processed and the logical storage space indicated to be updated is the logical storage space, for each unprocessed data update request, the locally stored second object mapping table is updated based on the unprocessed data update request, and the data to be updated indicated by the unprocessed data update request and the second object mapping table are persisted in accordance with the specified persistence method.

[0051] In this embodiment, the second object mapping table refers to the object mapping table with the latest creation time corresponding to the local logical storage space.

[0052] As for how to update the locally stored second object mapping table based on the unprocessed data update request, please refer to the relevant description in the above step 102, which will not be repeated here. As for how to persist the data to be updated indicated by the unprocessed data update request and the second object mapping table according to the specified persistence method, an example will be described below, which will not be repeated here.

[0053] Step 203: After the data indicated by each unprocessed data update request and the second object mapping table are persisted, a current snapshot of the logical storage space is created.

[0054] In this embodiment, as an embodiment, for any logical storage space on this node, the object mapping table corresponding to the logical storage space in the local node can be stored in a key-value (KV) storage method, wherein K can include the space identifier of the logical storage space (such as the volume identifier of the storage volume, the bucket identifier of the storage bucket, etc.), and the snapshot identifier of the snapshot of the logical storage space, etc., and V can include the object mapping table corresponding to the logical storage space, which is not specifically limited in this embodiment. Based on this, as an embodiment, after creating the current snapshot of the logical storage space, the second object mapping table can be used as the object mapping table corresponding to the current snapshot of the logical storage space, and at this time, the K in the KV indicating the second object mapping table in the local node can be updated from the space identifier of the logical storage space to the space identifier of the logical storage space and the snapshot identifier of the current snapshot of the logical storage space.

[0055] In this embodiment, after the second object mapping table is used as the object mapping table corresponding to the current snapshot of the logical storage space, a new object mapping table is created for the logical storage space.

[0056] Furthermore, after the new object mapping table is stored locally in this node, a first message indicating that the creation of this snapshot of the logical storage space has been completed is sent to the metadata node; if a second message indicating that the snapshot creation is successful is received from the metadata node within a set time period, it is determined that the creation of this snapshot of the logical storage space is successful, and after determining that the creation of this snapshot of the above-mentioned logical storage space is successful, the node is controlled to continue to receive data update requests; otherwise, it is determined that the creation of this snapshot of the logical storage space has failed, and the new object mapping table is deleted locally, and the K in the KV indicating the second object mapping table in the local node is updated from the space identifier of the logical storage space and the snapshot identifier of this snapshot of the logical storage space to the space identifier of the logical storage space. The above-mentioned second message is sent by the metadata node after receiving the first message sent by all storage nodes where the logical storage space is located.

[0057] In this embodiment, there is no need to wait for the object mapping tables of each storage node to be synchronized to other nodes such as metadata nodes and then uniformly persist them before creating the current snapshot of the logical storage space. Instead, the object mapping table can be persisted locally on the node, and the current snapshot of the logical storage space can be created after the object mapping table is persisted. This can effectively improve the efficiency of snapshot creation and improve data processing performance.

[0058] The above step 102 is further described below:

[0059] In this embodiment, as an embodiment, in order to ensure the recoverability of data, after the block usage status of the data storage block in the first object mapping table stored locally is updated based on the data update operation, the data to be updated indicated by the data update operation and the first object mapping table can be persisted in a specified persistence mode for subsequent data recovery. Specifically, for example, the data to be updated indicated by the data update operation and the first object mapping table can be synchronously written to the specified persistent storage medium to achieve data persistence; this embodiment is not specifically limited.

[0060] In this embodiment, the object mapping table used to indicate the block usage status of the data storage block on any storage node is stored locally on the storage node, so that the corresponding data can be persistently processed locally on the node without waiting for the object mapping table of each storage node to be synchronized to other nodes such as metadata nodes for unified persistence processing. This reduces frequent interactions between nodes and effectively improves the efficiency of data processing.

[0061] The following describes how to perform corresponding data processing when it is determined to start an incremental replication of data in any logical storage space to be replicated on the node:

[0062] In this embodiment, as an example, see Figure 3 As shown, when determining to start an incremental replication of data in any logical storage space to be replicated on the node, the method may further include the following steps:

[0063] Step 301: Create a snapshot of the logical storage space to be copied.

[0064] Step 302: after determining that the snapshot of the logical storage space to be copied is successfully created, obtain a third object mapping table and a fourth object mapping table corresponding to the logical storage space to be copied from a local computer.

[0065] The third object mapping table refers to the object mapping table corresponding to the latest snapshot of the logical storage space to be copied. The latest snapshot of the logical storage space to be copied here can be understood as the latest snapshot created in the snapshots of the logical storage space to be copied.

[0066] The fourth object mapping table refers to an object mapping table corresponding to a snapshot created during the last data replication of the logical storage space to be replicated.

[0067] In this embodiment, as described above, the object mapping table corresponding to the logical storage space is stored locally in accordance with the KV storage method, where K may include the space identifier of the logical storage space and the snapshot identifier of the snapshot of the logical storage space, etc.; based on this, the object mapping table corresponding to the latest snapshot of the logical storage space to be copied can be obtained locally as the third object mapping table, and the object mapping table corresponding to the snapshot created during the last data replication of the logical storage space to be copied can be obtained locally as the fourth object mapping table.

[0068] As for how to determine whether the snapshot of the logical storage space to be copied is successfully created in this step, please refer to the above related description, which will not be repeated here.

[0069] Step 303, for each data storage block belonging to the logical storage space to be copied on this node, check whether the block usage status of the data storage block in the third object mapping table is consistent with the block usage status of the data storage block in the fourth object mapping table; if not, determine that a data update operation has occurred on the data storage block since the last data replication of the logical storage space to be copied, and determine the data storage block as the data storage block to be copied; and after checking each data storage block belonging to the logical storage space to be copied on this node, incrementally copy the data in the logical storage space to be copied based on at least one determined data storage block to be copied.

[0070] In this embodiment, the incremental replication refers to the replication of only the data that has changed since the last data replication, such as the data that has been added, modified or deleted since the last data replication, during the data replication process. Therefore, by making a corresponding comparison between the third object mapping table and the fourth object mapping table, this embodiment can obtain the data storage blocks on this node that have undergone data changes since the last data replication of the logical storage space to be replicated, that is, the data storage blocks to be replicated; based on this, the incremental replication of the data in the logical storage space to be replicated can be achieved based on the at least one data storage block to be replicated that has been determined.

[0071] For example, as an embodiment, the at least one data storage block to be replicated determined above can realize incremental replication of the data in the logical storage space to be replicated. In the specific implementation, for example, it can be: this node will maintain a difference bitmap corresponding to the logical storage space to be replicated locally, and the difference bitmap is used to indicate whether each data storage block belonging to the logical storage space to be replicated on this node has undergone data changes since the last data replication of the logical storage space to be replicated. The difference bitmap can be determined based on the at least one data storage block to be replicated determined above; based on this, when incremental replication of the data in the logical storage space to be replicated is performed, the difference bitmap corresponding to the logical storage space to be replicated can be obtained first, so as to perform incremental replication of the data in the logical storage space to be replicated based on the difference bitmap.

[0072] In this embodiment, an object mapping table for indicating the block usage status of a data storage block on any storage node is stored locally on the storage node. In this way, when performing incremental replication of any logical storage space, the storage nodes where the logical storage space is located can concurrently and independently determine the data that has changed in the logical storage space since the last data replication, so as to perform incremental replication of the logical storage space. No additional cross-node interaction is required, thereby effectively improving the efficiency of data replication.

[0073] The following describes how to query and output the used storage space capacity using the object mapping table when receiving a third message indicating to query the used storage space capacity of any to-be-queried logical storage space:

[0074] In this embodiment, as an embodiment, when a third message indicating to query the used storage space capacity of any logical storage space to be queried is received, an object mapping table corresponding to the latest snapshot of the logical storage space to be queried is first obtained locally from the node;

[0075] Afterwards, the sum of the number of data storage blocks whose block usage status is the first status and the number of data storage blocks whose block usage status is the second status is determined based on the obtained object mapping table, so as to determine the space capacity of the used storage space in the storage space corresponding to each data storage block belonging to the logical storage space to be queried on this node based on the sum and the size of the data storage block of the logical storage space to be queried, and send a fourth message indicating the space capacity to the metadata node; here, the size of each data storage block in any logical storage space is the same;

[0076] Among them, after receiving the fourth message sent by all storage nodes where the logical storage space to be queried is located, the metadata node determines the used storage space capacity of the logical storage space to be queried according to the sum of the space capacities indicated by each fourth message and outputs it.

[0077] The object mapping table is further described below:

[0078] In this embodiment, as an embodiment, all data storage blocks of any logical storage space are grouped into multiple placement groups. Based on this, the above-mentioned creation of a new object mapping table for the logical storage space may specifically include: creating a new object mapping table for each placement group of the logical storage space; wherein any object mapping table corresponding to any placement group is used to indicate the block usage status of at least one data storage block belonging to the placement group on the node within the specified time period corresponding to the object mapping table.

[0079] In this embodiment, based on the above description, it can be known that the object mapping table corresponding to the logical storage space is stored locally in accordance with the KV storage method, where K may include the space identifier of the logical storage space and the snapshot identifier of the snapshot of the logical storage space; based on this, K may also include the group identifier of the placement group.

[0080] For any placement group of any logical storage space on this node, the specified operation of any object mapping table corresponding to the placement group is independent of the specified operation of any object mapping table corresponding to other placement groups; wherein the specified operation at least includes access, update, and persistence, etc., which is not specifically limited in this embodiment.

[0081] In this embodiment, by further refining the K in KV to the placement group level, when performing operations such as data access, data update, and data persistence in the logical storage space, the specified operations (such as access, update, or persistence, etc.) of the object mapping table of each placement group corresponding to the logical storage space can be executed concurrently, that is, they can be independent of each other and have no effect on each other, which can further improve the efficiency of data processing.

[0082] At this point, the description of the method provided in the embodiment of the present application is completed. The following is a description of the device provided in the embodiment of the present application:

[0083] See also Figure 4 , Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 4 As shown, the data processing device 400 includes a creation module 401 and an update module 402;

[0084] Wherein, the creation module 401 is used to create a new object mapping table for the logical storage space and store the new object mapping table locally at the node when detecting a creation indication of using at least one data storage block on the node as a data storage block of any logical storage space to be created, or when determining to start a snapshot creation of any logical storage space on the node; any object mapping table corresponding to any logical storage space stored locally at the node is used to indicate the block usage status of at least one data storage block belonging to the logical storage space on the node within the specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table;

[0085] Update module 402 is used to update the block usage status of the data storage block in the first object mapping table stored locally based on the data update operation when any data update operation is detected to occur in any data storage block on this node; the first object mapping table refers to the object mapping table with the latest creation time of the mapping table in the object mapping table corresponding to the logical storage space to which the data storage block belongs.

[0086] As an embodiment, after updating the block usage status of the data storage block in the locally stored first object mapping table based on the data update operation, the device further includes: a persistence module, which is used to persist the data to be updated indicated by the data update operation and the first object mapping table in accordance with a specified persistence method.

[0087] As an embodiment, the creation module 401 is further used to: when it is determined to start creating a snapshot of any logical storage space on the node, control the node to suspend receiving data update requests;

[0088] When there is currently at least one unprocessed data update request that has been received but not yet processed and the indicated logical storage space to be updated is the logical storage space, for each unprocessed data update request, update the locally stored second object mapping table based on the unprocessed data update request, and perform persistence processing on the data to be updated indicated by the unprocessed data update request and the second object mapping table in accordance with the specified persistence method; the second object mapping table refers to the object mapping table with the latest creation time corresponding to the logical storage space;

[0089] After the data indicated by each unprocessed data update request and the second object mapping table are persisted, a current snapshot of the logical storage space is created.

[0090] As an embodiment, the creation module 401 is further used to: after the creation of the current snapshot of the logical storage space is completed, record the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space;

[0091] Among them, after the second object mapping table is used as the object mapping table corresponding to the current snapshot of the logical storage space and recorded, a new object mapping table is created for the logical storage space; and after the new object mapping table is stored locally in this node, a first message indicating that the current snapshot of the logical storage space has been created is sent to the metadata node. If the second message indicating that the snapshot creation is successful is received from the metadata node within the set time period, it is determined that the current snapshot of the logical storage space is created successfully; otherwise, it is determined that the current snapshot of the logical storage space has failed to be created, and the new object mapping table is deleted from the local; the second message is sent by the metadata node after receiving the first message sent by all storage nodes where the logical storage space is located.

[0092] As an embodiment, the device further includes: a replication module, which is used to create a snapshot of the logical storage space to be replicated when it is determined to start an incremental replication of data in any logical storage space to be replicated on the node;

[0093] After determining that the snapshot of the logical storage space to be copied is successfully created, a third object mapping table and a fourth object mapping table corresponding to the logical storage space to be copied are obtained locally; wherein the third object mapping table refers to the object mapping table corresponding to the latest snapshot of the logical storage space to be copied; and the fourth object mapping table refers to the object mapping table corresponding to the snapshot created during the last data replication of the logical storage space to be copied;

[0094] For each data storage block on the current node that belongs to the logical storage space to be replicated, check whether the block usage status of the data storage block in the third object mapping table is consistent with the block usage status of the data storage block in the fourth object mapping table; if not, determine that a data update operation has occurred on the data storage block since the last data replication of the logical storage space to be replicated, and determine the data storage block as the data storage block to be replicated; and after checking each data storage block on the current node that belongs to the logical storage space to be replicated, incrementally replicate the data in the logical storage space to be replicated based on at least one determined data storage block to be replicated.

[0095] As an embodiment, all data storage blocks of any logical storage space are grouped into a plurality of placement groups;

[0096] Creating a new object mapping table for the logical storage space includes: creating a new object mapping table for each placement group of the logical storage space; wherein, any object mapping table corresponding to any placement group is used to indicate the block usage status of at least one data storage block belonging to the placement group on the node within a specified time period corresponding to the object mapping table; for any placement group of any logical storage space on the node, the specified operation of any object mapping table corresponding to the placement group is independent of the specified operation of any object mapping table corresponding to other placement groups; the specified operation at least includes access, update, and persistence.

[0097] So far, completed Figure 4 Structural description of the device shown.

[0098] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0099] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0100] See also Figure 5, is a schematic diagram of the hardware structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device may include a processor 501, a communication interface 502, a computer-readable storage medium 503 and a communication bus 504. The processor 501, the communication interface 502 and the computer-readable storage medium 503 communicate with each other through the communication bus 504. Among them, computer program instructions are stored on the computer-readable storage medium 503; the processor 501 can execute the steps of the method described in the above embodiment by executing the computer program instructions stored on the computer-readable storage medium 503. The electronic device may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0101] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, on which a number of computer program instructions are stored. When the computer program instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0102] Exemplarily, the computer-readable storage medium may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the computer-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[0103] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method is applied to any storage node in a distributed storage system; the method comprises: When a creation indication of using at least one data storage block on the node as a data storage block of any logical storage space to be created is detected, or when it is determined to start a snapshot creation of any logical storage space on the node, a new object mapping table is created for the logical storage space, and the new object mapping table is stored locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate the block usage status of at least one data storage block belonging to the logical storage space on the node within the specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table; When any data update operation is detected in any data storage block on this node, the block usage status of the data storage block in the locally stored first object mapping table is updated based on the data update operation; the first object mapping table refers to the object mapping table with the latest creation time in the object mapping table corresponding to the logical storage space to which the data storage block belongs.

2. The method according to claim 1, characterized in that: After updating the block usage status of the data storage block in the locally stored first object mapping table based on the data update operation, the method further includes: The data to be updated indicated by the data update operation and the first object mapping table are persisted in accordance with a specified persistence mode.

3. The method according to claim 1, characterized in that When determining to start creating a snapshot of any logical storage space on the node, the method further includes: Control this node to stop receiving data update requests; When there is currently at least one unprocessed data update request that has been received but not yet processed and the indicated logical storage space to be updated is the logical storage space, for each unprocessed data update request, update the locally stored second object mapping table based on the unprocessed data update request, and perform persistence processing on the data to be updated indicated by the unprocessed data update request and the second object mapping table in accordance with the specified persistence method; the second object mapping table refers to the object mapping table with the latest creation time corresponding to the logical storage space; After the data indicated by each unprocessed data update request and the second object mapping table are persisted, a current snapshot of the logical storage space is created.

4. The method according to claim 3, characterized in that The method further includes: after creating the current snapshot of the logical storage space, using the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space; Among them, after using the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space, a new object mapping table is created for the logical storage space; and after storing the new object mapping table locally in this node, a first message indicating that the current snapshot of the logical storage space has been created is sent to the metadata node. If a second message indicating that the snapshot creation is successful is received from the metadata node within a set time period, it is determined that the current snapshot of the logical storage space is created successfully; otherwise, it is determined that the current snapshot of the logical storage space has failed to be created, and the new object mapping table is deleted from the local; the second message is sent by the metadata node after receiving the first message sent by all storage nodes where the logical storage space is located.

5. The method according to claim 4, characterized in that The method further comprises: When it is determined to start an incremental replication of data in any logical storage space to be replicated on the node, a snapshot of the logical storage space to be replicated is created; After determining that the snapshot of the logical storage space to be copied is successfully created, a third object mapping table and a fourth object mapping table corresponding to the logical storage space to be copied are obtained locally; wherein the third object mapping table refers to the object mapping table corresponding to the latest snapshot of the logical storage space to be copied; and the fourth object mapping table refers to the object mapping table corresponding to the snapshot created during the last data replication of the logical storage space to be copied; For each data storage block on this node that belongs to the logical storage space to be copied, check whether the block usage status of the data storage block in the third object mapping table is consistent with the block usage status of the data storage block in the fourth object mapping table; if not, determine that a data update operation has occurred on the data storage block since the last data replication of the logical storage space to be copied, and determine the data storage block as the data storage block to be copied; and after checking each data storage block on this node that belongs to the logical storage space to be copied, incrementally copy the data in the logical storage space to be copied based on at least one determined data storage block to be copied.

6. The method according to claim 1, characterized in that All data storage blocks of any logical storage space are grouped into multiple placement groups; The creating of a new object mapping table for the logical storage space comprises: creating a new object mapping table for each placement group of the logical storage space; wherein, any object mapping table corresponding to any placement group is used to indicate the block usage status of at least one data storage block belonging to the placement group on the node within the specified time period corresponding to the object mapping table; for any placement group of any logical storage space on the node, the specified operation of any object mapping table corresponding to the placement group is independent of the specified operation of any object mapping table corresponding to other placement groups; the specified operation at least includes access, update, and persistence.

7. A data processing device, characterized in that: The device is applied to any storage node in a distributed storage system; the device comprises: A creation module, used for, when detecting a creation indication of using at least one data storage block on the node as a data storage block of any logical storage space to be created, or when determining to start a snapshot creation of any logical storage space on the node, creating a new object mapping table for the logical storage space, and storing the new object mapping table locally on the node; any object mapping table corresponding to any logical storage space stored locally on the node is used to indicate the block usage status of at least one data storage block belonging to the logical storage space on the node within a specified time period corresponding to the object mapping table; the specified time period corresponding to any object mapping table is determined according to the creation time of the object mapping table; An update module is used to update the block usage status of the data storage block in the first object mapping table stored locally based on the data update operation when any data update operation is detected to occur in any data storage block on this node; the first object mapping table refers to the object mapping table with the latest creation time in the object mapping table corresponding to the logical storage space to which the data storage block belongs.

8. The device according to claim 7, characterized in that After updating the block usage status of the data storage block in the locally stored first object mapping table based on the data update operation, the device further includes: a persistence module, used to perform persistence processing on the data to be updated indicated by the data update operation and the first object mapping table in accordance with a specified persistence mode; and / or, The creation module is further used to: when it is determined to start creating a snapshot of any logical storage space on the node, control the node to suspend receiving data update requests; when there is currently at least one unprocessed data update request that has been received but not processed and the indicated logical storage space to be updated is the logical storage space, for each unprocessed data update request, update the locally stored second object mapping table based on the unprocessed data update request, and perform persistence processing on the data to be updated indicated by the unprocessed data update request and the second object mapping table in accordance with the specified persistence method; the second object mapping table refers to the object mapping table with the latest creation time corresponding to the logical storage space; after the data indicated by each unprocessed data update request and the second object mapping table are all persistently processed, create the current snapshot of the logical storage space; and / or, The creation module is further used to: after creating the current snapshot of the logical storage space, use the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space; wherein, after using the second object mapping table as the object mapping table corresponding to the current snapshot of the logical storage space, a new object mapping table is created for the logical storage space; and after storing the new object mapping table locally in this node, a first message indicating that the current snapshot of the logical storage space has been created is sent to the metadata node; if a second message indicating that the snapshot has been created successfully and sent by the metadata node is received within a set time period, it is determined that the current snapshot of the logical storage space has been created successfully; otherwise, it is determined that the current snapshot of the logical storage space has failed to be created, and the new object mapping table is deleted from the local; the second message is sent by the metadata node after receiving the first message sent by all storage nodes where the logical storage space is located; and / or, The device further includes: a replication module, which is used to create a snapshot of the logical storage space to be replicated when it is determined to start an incremental replication of data in any logical storage space to be replicated on the node; after determining that the snapshot of the logical storage space to be replicated is successfully created, obtain a third object mapping table and a fourth object mapping table corresponding to the logical storage space to be replicated from the local; wherein the third object mapping table refers to the object mapping table corresponding to the latest snapshot of the logical storage space to be replicated; the fourth object mapping table refers to the object mapping table corresponding to the snapshot created when the logical storage space to be replicated was replicated last time; for For each data storage block on this node that belongs to the logical storage space to be copied, check whether the block usage status of the data storage block in the third object mapping table is consistent with the block usage status of the data storage block in the fourth object mapping table; if not, determine that a data update operation has occurred on the data storage block since the last data replication of the logical storage space to be copied, and determine the data storage block as the data storage block to be copied; and after checking each data storage block on this node that belongs to the logical storage space to be copied, incrementally copy the data in the logical storage space to be copied based on at least one determined data storage block to be copied.

9. The device according to claim 7, characterized in that All data storage blocks of any logical storage space are grouped into multiple placement groups; The creating of a new object mapping table for the logical storage space comprises: creating a new object mapping table for each placement group of the logical storage space; wherein, any object mapping table corresponding to any placement group is used to indicate the block usage status of at least one data storage block belonging to the placement group on the node within the specified time period corresponding to the object mapping table; for any placement group of any logical storage space on the node, the specified operation of any object mapping table corresponding to the placement group is independent of the specified operation of any object mapping table corresponding to other placement groups; the specified operation at least includes access, update, and persistence.

10. An electronic device, characterized in that: The electronic device includes: Processor; and A computer-readable storage medium, wherein computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are executed by the processor, the processor executes the steps in any one of the methods of claims 1 to 6.

Images