Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and the method comprises the steps: determining a target file corresponding to to-be-updated data, and a target data block group; when metadata information of a target data block group and information of a management node of the target data block group are queried and the storage node is the management node of the target data block group, determining a locking state corresponding to the to-be-updated data according to locking information recorded in a memory; under the condition that the locking state is an unlocked state, updating the locking state to a locked state, and executing corresponding data updating processing according to the data updating request; and when the locking state is a locked state, refusing to execute corresponding data updating processing according to the data updating request. According to the method, the data processing performance can be improved under the condition that the data consistency is ensured.

Description

Technical Field

[0001] The present application relates to the field of cloud storage technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Art

[0002] With the development of society, safety has become a focus of people's attention. The popularity of security monitoring has provided a certain guarantee for people's safe life.

[0003] There is a large amount of video data in security monitoring, and cloud storage is currently commonly used to store this video data.

[0004] In order to achieve high availability of the video data storage cluster, a fully symmetric architecture is usually used to store video data, that is, each node in the cluster system plays an equal role and jointly manages and maintains metadata.

[0005] How to ensure data consistency in a fully symmetric cloud storage architecture has become a hot research direction. Summary of the Invention

[0006] In view of this, the present application provides a data processing method, apparatus, device and storage medium.

[0007] According to a first aspect of an embodiment of the present application, there is provided a data processing method, applied to a storage node in a fully symmetric cloud storage architecture, the method comprising:

[0008] In the case of receiving a data update request, determining a target file corresponding to the data to be updated, and a target data block group in the target file corresponding to the data to be updated;

[0009] When metadata information of the target data block group and information of a target data block group management node corresponding to the target data block group are queried from a cluster database, determining the target data block group management node according to the information of the target data block group management node;

[0010] In a case where the storage node serves as the target data block group management node, determining a locking state corresponding to the data to be updated according to the locking information recorded in the memory;

[0011] When the lock state corresponding to the data to be updated is unlocked, the lock state corresponding to the data to be updated is updated to locked, and corresponding data update processing is performed according to the data update request;

[0012] When the lock state corresponding to the data to be updated is locked, the corresponding data update process is refused to be executed according to the data update request.

[0013] According to a second aspect of an embodiment of the present application, there is provided a data processing device deployed on a storage node in a fully symmetric cloud storage architecture, the device comprising:

[0014] a determining unit configured to, upon receiving a data update request, determine a target file corresponding to the data to be updated and a target data block group in the target file corresponding to the data to be updated;

[0015] The determining unit is further configured to determine the target data block group management node according to the information of the target data block group management node when metadata information of the target data block group and information of the target data block group management node corresponding to the target data block group are queried from the cluster database;

[0016] The determining unit is further configured to determine, when the storage node serves as the target data block group management node, a locking state corresponding to the data to be updated based on the locking information recorded in the memory;

[0017] a processing unit configured to update the locking state corresponding to the data to be updated to locked when the locking state corresponding to the data to be updated is unlocked, and perform corresponding data update processing according to the data update request;

[0018] The processing unit is further configured to refuse to execute the corresponding data update processing according to the data update request when the locking state corresponding to the data to be updated is locked.

[0019] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0020] Memory for storing computer programs;

[0021] The processor is used to implement the method provided in the first aspect when executing the program stored in the memory.

[0022] According to a fourth aspect of an embodiment of the present application, a non-temporary computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0023] The data processing method of the embodiment of the present application allocates a data block group management node to each data block group in the file, and stores the metadata information of the data block group and the metadata information of the data block group management node in the cluster database. When any storage node in the cloud storage cluster receives a data update request, it can query the cluster database for the metadata information of the corresponding target data block group and the target data block group management node, and when the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are queried from the cluster database, the target data block group management node is determined based on the information of the target data block group management node. When the storage node serves as the target data block group management node, it can determine the locking status corresponding to the data to be updated based on the locking information recorded in the memory. When the locking status corresponding to the data to be updated is unlocked, the locking status corresponding to the data to be updated is updated to locked, and the corresponding data update processing is executed according to the data update request; when the locking status corresponding to the data to be updated is locked, the corresponding data update processing is refused to be executed according to the data update request. By allocating a data block group management node to the data block group, the data block group management node records the locking information related to the data block group through the memory, thereby improving the data processing performance while ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of an EC block group provided in an embodiment of the present application;

[0026] Figure 3 This is a flowchart of allocating managers provided by an embodiment of the present application;

[0027] Figure 4 This is a flow chart of an ECG locking process provided by an embodiment of the present application;

[0028] Figure 5 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0029] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some technical terms involved in the embodiments of the present application are briefly explained below.

[0031] 1. Database (DB): refers to a collection of related, structured data stored on a computer's storage device. A database contains various components, including tables, views, stored procedures, records, fields, indexes, etc.

[0032] 2. Video streaming: refers to the transmission of video data, which can be processed as a stable and continuous stream through the network.

[0033] 3. Data Storage Device (DSD): This stands for storage node and is the module in an object storage system that reads and writes object data. A DSD stores data on disk tracks and sectors, groups several tracks and sectors into objects, and provides external access to data through these objects.

[0034] 4. MDS (Management Data System): A management node in a data storage system that stores object index information, including the object name, the specific location where the object data is stored, and the last modification time of the object.

[0035] 5. Clustering: A cluster is a group of independent computers connected via a high-speed network. These computers form a group and are managed as a single system. When a client interacts with the cluster, the cluster behaves like a standalone server. Clustering is used to improve availability and scalability.

[0036] 6. Domain: Cloud storage can manage multiple storage nodes. Resource pool data created within a domain will only be written to storage resources within the domain, achieving storage resource isolation by domain.

[0037] 7. Full Symmetry: Cluster file systems can be categorized as symmetric or asymmetric based on how metadata is managed. In a symmetric cluster system, each node plays an equal role, jointly managing and maintaining metadata. Nodes synchronize information and perform operations such as mutual exclusion locks over a high-speed network. In an asymmetric cluster file system, one or more nodes are dedicated to metadata management, and other nodes must frequently communicate with the metadata node to obtain the latest metadata, such as directory listings and file attributes.

[0038] 8. EC (Erasure Coding) Block Group (ECG): An EC redundancy group consisting of multiple blocks. For example, if the block size is 8MB, EC is 4+2, and the unit is 1MB. 6 blocks form an ECG, which contains 8 4+2 stripes with 1MB units. Each block is divided into 8 units (u0, u1, to u7) in sequence, and the ux of each block forms a 4+2 stripe.

[0039] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0040] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0041] See Figure 1 , is a flow chart of a data processing method provided in an embodiment of the present application. For example, the data processing method can be applied to storage nodes in a fully symmetric cloud storage architecture, such as Figure 1 As shown, the data processing method may include the following steps:

[0042] Step S100: upon receiving a data update request, determining a target file corresponding to the data to be updated and a target data block group in the target file corresponding to the data to be updated.

[0043] In the embodiment of the present application, a user may send a data update request to the cloud storage cluster through a client, requesting to update data in a specific data block group in a specific file.

[0044] Exemplarily, data updating includes initial data writing or updating of already written data.

[0045] Upon receiving a data update request, any storage node in the cloud storage cluster can determine the file corresponding to the data to be updated (which may be called the target file) and the data block group in the target file corresponding to the data to be updated (which may be called the target data block group).

[0046] Exemplarily, a file may include multiple data block groups, such as EC block groups, and different data block groups correspond to different offsets in the file.

[0047] The storage node may determine the data block group corresponding to the data to be updated based on the offset corresponding to the data to be updated in the file.

[0048] Step S110 : When metadata information of the target data block group and information of the target data block group management node corresponding to the target data block group are queried from the cluster database, the target data block group management node is determined according to the information of the target data block group management node.

[0049] In an embodiment of the present application, in order to improve data processing performance, a file management node can be assigned to the files created in the cloud storage cluster, and a data block group management node can be assigned to the data block groups created in the files. The data block group management node is used to process data updates of the corresponding data block groups.

[0050] Exemplarily, the file management node may be allocated by a master node of the cloud storage cluster.

[0051] For any file, the data block group management node of each data block group in the file can be allocated by the file management node of the file.

[0052] For example, in order to avoid conflicts in the allocation of data block group management nodes and ensure the consistency of the allocation of data block group management nodes, the file management node can allocate data block group management nodes through memory mutual exclusion. That is, for a data block group, a thread of the file management node can record the locking information for the data block group in the memory during the process of allocating a data block group management node for the data block group, so as to avoid other threads of the file management node allocating different data block group management nodes to the same data block group.

[0053] Similarly, when the cluster master node allocates file management nodes to files in the cloud storage cluster, it can also use memory mutual exclusion to ensure the consistency of file management node allocation.

[0054] Exemplarily, the master node of the cloud storage cluster is determined by dynamic negotiation among the storage nodes in the cloud storage cluster.

[0055] For example, the storage node with the largest IP address in the cloud storage cluster can be determined as the master node; wherein, in the event of a failure of the current master node, the storage node with the largest IP address among the remaining available storage nodes can be automatically determined as the new master node.

[0056] Exemplarily, when a file management node creates a data block group and assigns a data block group management node to the data block group, metadata information of the data block group and information of the corresponding data block group management node (such as identification information of the data block management node, for example, an IP address) can be stored in the cluster database.

[0057] Exemplarily, the metadata information of the data block group includes part or all of the information such as the block group ID (identifier) ​​and the location of the data block in the disk.

[0058] Exemplarily, for an EC block group, the metadata information of the data block group may further include the number of data blocks and check blocks in the data block group.

[0059] Accordingly, when the storage node determines the target data block group corresponding to the data to be updated, it can query the cluster database for metadata information of the target data block group and the data block group management node corresponding to the target data block group (which can be called the target data block group management node).

[0060] When the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are queried from the cluster database, that is, the target data block group already exists and the target data block group management node also already exists, in this case, the target data block group management node can be determined based on the information of the target data block group management node.

[0061] Step S120: When the storage node serves as the target data block group management node, the locking state corresponding to the data to be updated is determined according to the locking information recorded in the memory.

[0062] In an embodiment of the present application, in order to improve data processing performance while ensuring data consistency, the data block group management node may record relevant locking information of the managed data block group in memory (the memory of this management node).

[0063] Accordingly, when the target data block group management node receives a data update request, it can determine the locking status corresponding to the data to be updated based on the locking information recorded in the memory.

[0064] Exemplarily, the locking status corresponding to the data to be updated may include locked or unlocked.

[0065] Exemplarily, the lock state corresponding to the data to be updated is locked, indicating that the corresponding data is currently being updated; the lock state corresponding to the data to be updated is unlocked, indicating that the corresponding data is currently not being updated.

[0066] It should be noted that in an embodiment of the present application, if the current storage node (the storage node that receives the data update request in step S100) is not the target data block group management node, the storage node can send the data update request to the determined target data block group management node.

[0067] For example, the specific processing flow when the target data block group management node receives a data update request is similar to the processing flow when the storage node acts as the target data block group management node (see the relevant descriptions in steps 120 to 140), and the embodiments of the present application will not be repeated here.

[0068] Step S130 : When the lock state corresponding to the data to be updated is unlocked, the lock state corresponding to the data to be updated is updated to locked, and corresponding data update processing is performed according to the data update request.

[0069] Step S140: When the lock state corresponding to the data to be updated is locked, refuse to execute the corresponding data update process according to the data update request.

[0070] In an embodiment of the present application, when the target data block group management node determines that the locking status corresponding to the data to be updated is unlocked, that is, the corresponding data is not currently in an update state, in this case, the target data block group management node can update the locking status corresponding to the data to be updated to locked, and perform corresponding data update processing based on the data update request.

[0071] When the target data block group management node determines that the lock status corresponding to the data to be updated is locked, that is, the corresponding data is currently in the update state, in this case, in order to avoid data update conflicts and ensure data consistency, the target data block group management node can refuse to perform the corresponding data update processing based on the data update request.

[0072] It can be seen that in Figure 1 In the method flow shown, by allocating a data block group management node to each data block group in the file, and storing the metadata information of the data block group and the metadata information of the data block group management node in the cluster database, any storage node in the cloud storage cluster can query the corresponding target data block group and the metadata information of the target data block group management node in the cluster database when receiving a data update request, and when the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are queried from the cluster database, the target data block group management node is determined based on the information of the target data block group management node; in this storage section When the point is used as the target data block group management node, the locking status corresponding to the data to be updated can be determined based on the locking information recorded in the memory. When the locking status corresponding to the data to be updated is unlocked, the locking status corresponding to the data to be updated is updated to locked, and the corresponding data update processing is executed according to the data update request; when the locking status corresponding to the data to be updated is locked, the corresponding data update processing is refused to be executed according to the data update request. By allocating a data block group management node to the data block group, the data block group management node records the locking information related to the data block group through the memory, thereby improving the data processing performance while ensuring data consistency.

[0073] In some embodiments, the data processing method provided in the embodiments of the present application may further include:

[0074] If metadata information of the target data block group is not found from the cluster database, but metadata information of the target file and information of the target file management node corresponding to the target file are found, determining the target file management node based on the information of the target file management node;

[0075] When the storage node serves as the target file management node, a target data block group is created in the target file, a target data block group management node is assigned to the target data block group, and metadata information of the target data block group and information of the target data block group management node are stored in the cluster database;

[0076] When the allocated target data block group management node is not the current storage node, the data update request is sent to the target data block group management node.

[0077] Exemplarily, when the cluster master node creates a file and assigns a file management node, it can store the file's metadata information and the corresponding file management node information (such as the file management node's identification information, for example, the IP address) in the cluster database.

[0078] If the storage node fails to find metadata information of the target data block group from the cluster database, it may further query the cluster database for metadata information of the target file and the file management node corresponding to the target file (which may be called the target file management node).

[0079] When the metadata information of the target file and the information of the target file management node corresponding to the target file are queried from the cluster database, that is, the target file already exists and the target file management node also exists, in this case, the target file management node can be determined based on the information of the target file management node corresponding to the target file.

[0080] In the case where the storage node is a target file management node, the storage node may create a target data block group in the target file and allocate a target data block group management node to the target data block group.

[0081] Exemplarily, the target file management node may be randomly selected from the storage nodes in the cloud storage cluster to select the target data block group management node, or the storage node that initially receives the data update request may be preferentially selected as the target data block group management node.

[0082] It should be noted that in order to ensure the availability of the data block group management node, offline nodes need to be excluded during the data block group management node selection process; in addition, in order to avoid excessive load on the data block group management node, storage nodes whose CPU usage exceeds the preset usage threshold, or whose memory usage exceeds the preset usage threshold, can also be excluded during the data block group management node selection process.

[0083] Exemplarily, the target file management node may store metadata information of the target data block group and information of the target data block group management node in the cluster database.

[0084] Exemplarily, when the assigned target data block group management node is not the current storage node (the storage node serving as the target file management node), the target file management node may send a data update request to the target data block group management node.

[0085] The processing flow after the target data block group management node receives the data update request can be referred to the relevant description in the above embodiment, and will not be repeated in the embodiment of the present application.

[0086] It should be noted that, when the allocated target data block group management node is the storage node, the storage node may perform relevant processing in the manner described in steps S120 to S140, which will not be repeated here in the embodiment of the present application.

[0087] In some embodiments, the data processing method provided in the embodiments of the present application may further include:

[0088] If no metadata information of the target file is found in the cluster database and the current storage node is the cluster master node, the target file is created, a target file management node is assigned to the target file, the metadata information of the target file and the information of the target file management node are stored in the cluster database, and a data update request is sent to the target file management node; the cluster master node is determined by dynamic negotiation among the storage nodes in the cluster;

[0089] In the case that the assigned target file management node is not the current storage node, the data update request is sent to the assigned target file management node.

[0090] Exemplarily, when the storage node fails to find metadata information of the target file from the cluster database, it may determine that the target file has not been created. In this case, a target file creation process needs to be performed.

[0091] When a storage node serves as a cluster master node, the storage node can create a target file and allocate a target file management node to the target file.

[0092] Exemplarily, the cluster master node may select the target file management node from the storage nodes in the cloud storage cluster by random selection, or may preferentially select the storage node that initially receives the data update request as the target file management node.

[0093] It should be noted that in order to ensure the availability of file management nodes, offline nodes need to be excluded during the file management node selection process; in addition, in order to avoid excessive load on file management nodes, storage nodes whose CPU usage exceeds the preset usage threshold, or whose memory usage exceeds the preset usage threshold, can also be excluded during the file management node selection process.

[0094] Exemplarily, the cluster master node may store metadata information of the target file and information of the target file management node in the cluster database.

[0095] It should be noted that in an embodiment of the present application, when the current storage node (i.e., the storage node that receives the data update request in step S100) is not the cluster master node, the data update request can be sent to the queried cluster master node.

[0096] Among them, the specific processing flow when the cluster master node receives the data update request sent by the storage node is similar to the processing flow when the storage node acts as the cluster master node (see the relevant description in the above embodiment), and the embodiments of this application will not be repeated here.

[0097] Exemplarily, when the assigned target file management node is not the current storage node (the storage node that serves as the cluster master node), the cluster master node may send a data update request to the target file management node.

[0098] When the target file management node receives a data update request, it can perform target data block group creation processing. The specific processing flow can be found in the relevant description in the above embodiment, and the embodiment of the present application will not repeat it.

[0099] When the target file management node completes the creation of the target data block group and the allocation of the target data block group management node, the data update request may be sent to the target data block group management node.

[0100] The processing flow after the target data block group management node receives the data update request can be referred to the relevant description in the above embodiment, and will not be repeated in the embodiment of the present application.

[0101] It should be noted that when the assigned target file management node is the current storage node, the storage node can create the target data block group and allocate the target data block group management node in the manner described in the above embodiments. The specific implementation can refer to the relevant description in the above embodiments, and the embodiments of this application will not be repeated here.

[0102] In some embodiments, the data update request may include a locking strategy, which may include locking the entire data block group or stripe locking; a data block group includes multiple stripes;

[0103] The lock status corresponding to the above-mentioned data to be updated may include:

[0104] In the case where the locking strategy is to lock the entire data block group, if the locking state of the target data block group is locked, the locking state corresponding to the data to be updated is determined to be locked;

[0105] For the case where the locking strategy is stripe locking, when the locking status of at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking status corresponding to the data to be updated is determined to be locked; when the locking status of the stripes corresponding to the data to be updated in the target data block group are all unlocked, the locking status corresponding to the data to be updated is determined to be unlocked.

[0106] For example, to improve the flexibility of locking, it is no longer limited to locking in stripes, but the entire data block group can be locked as needed, or one or more stripes in the data block group can be locked.

[0107] Illustratively, a data block group includes multiple stripes.

[0108] For example, taking the EC block group as an example, Figure 2 As shown in the figure, assume that an EC block group includes 6 data blocks, each data block is divided into 8 units (assuming they are u0 to u7), and the ux (x = 0, 1, ..., 7) of each block forms a 4+2 stripe. For example, the u0 of each block forms a 4+2 stripe.

[0109] Accordingly, when initiating a data update request through the client, the user can also select a locking strategy as needed, and carry the locking strategy in the data update request and send it to the storage node of the cloud storage cluster.

[0110] When a storage node receives a data update request, it may obtain a lock request included in the data update request.

[0111] For example, if the locking strategy is to lock the entire data block group, when the locking state of the target data block group is locked, it is determined that the locking state corresponding to the data to be updated is locked;

[0112] For the case where the locking strategy is stripe locking, when the locking status of at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking status corresponding to the data to be updated is determined to be locked; when the locking status of the stripes corresponding to the data to be updated in the target data block group are all unlocked, the locking status corresponding to the data to be updated is determined to be unlocked.

[0113] In one example, when the data update request does not include a locking strategy, it is determined that the locking strategy is to lock the entire data block group.

[0114] For example, in order to optimize locking performance and improve locking efficiency, the default locking strategy may lock the entire data block group.

[0115] Accordingly, when the data update request does not include a locking strategy, the locking strategy is determined to be locking the entire data block group.

[0116] In some embodiments, after updating the lock status corresponding to the to-be-updated data to locked, the following steps may also be performed:

[0117] When the corresponding data update processing is completed according to the data update request, or when the preset lock timeout period is reached, the lock state corresponding to the data to be updated is updated to unlocked.

[0118] For example, to avoid deadlock or lock leakage, unlocking can support active release and timeout release mechanisms.

[0119] Accordingly, the data block group management node can actively perform unlocking processing when completing the corresponding data update processing according to the data update request.

[0120] For example, the data block group management node may explicitly call a lock release interface to release the lock and update the locking status of the corresponding data from locked to unlocked.

[0121] In addition, the lock timeout period can be pre-set, such as 2 minutes. When the data block group management node determines that the pre-set lock timeout period has been reached, the lock release process must be performed regardless of whether the data update process is completed, and the lock status of the corresponding data must be updated from locked to unlocked.

[0122] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0123] In an embodiment of the present application, in order to improve locking efficiency while ensuring data consistency, the concept of file management node / data block group (taking EC block group (ECG) as an example) management node is proposed, and ECG-level locking granularity is provided.

[0124] Each file and ECG is assigned a corresponding management node. All actions, such as creating ECGs under the corresponding file and locking and updating ECGs, are handled by the corresponding management node. Concurrent and mutually exclusive actions are performed in the management node's memory, ensuring locking consistency and avoiding the use of distributed locks. Every node in the cluster can be selected as a management node, eliminating single-node performance bottlenecks and single-node failures.

[0125] In addition, the program acquires locks at the ECG level by default, improving locking efficiency. The system's read and write update strategy maximizes the guarantee that updates to the same ECG are initiated by the same client, reducing stripe-level locking.

[0126] The specific processing flow is described below.

[0127] In this embodiment, the data processing flow may include two parts: assigning a manager and ECG locking.

[0128] 1. Assign managers.

[0129] For example, Figure 3 As shown, the process of allocating managers may include: the cluster master node allocating the file management node; and the file management node allocating the ECG management node.

[0130] 1.1. The cluster master node allocates file management nodes.

[0131] Exemplarily, each storage node in the cloud storage cluster may determine the cluster master node through dynamic negotiation.

[0132] For any file, during the process of creating the file, the cluster master node can allocate a file management node for the file.

[0133] For example, to ensure consistency in file creation, the cluster master node may create files through memory mutual exclusion to avoid concurrent creation of two metadata indexes for a file.

[0134] For example, in order to ensure consistency in the allocation of file management nodes, the cluster master node may allocate file management nodes through memory mutual exclusion.

[0135] Exemplarily, the cluster master node may select the file management node by randomly selecting the file management node from the storage nodes in the cloud storage cluster.

[0136] Exemplarily, when the cluster master node completes file creation and file management node allocation, the file metadata information and file management node information may be recorded in the cluster database.

[0137] 1.2. File management nodes are assigned ECG management nodes.

[0138] For example, a file may include multiple ECGs, and updating of data in the file needs to locate the corresponding ECG according to the offset in the file.

[0139] The number of ECGs included in a file can be configured according to requirements. For example, a single file can be limited to a maximum of 50 ECGs.

[0140] In order to ensure the consistency of data updates, the ECG needs to be locked first during the update of the corresponding ECG data.

[0141] In order to avoid the locking of ECG logic involving distributed locks, a management node can be assigned to ECG.

[0142] Exemplarily, an ECG management node may be assigned to each ECG by the file management node.

[0143] Exemplarily, the file management node may allocate ECG management nodes through memory mutual exclusion to ensure consistency of ECG management node allocation.

[0144] In addition, the file management node may record the information of the ECG management node and the ECG metadata information together into the cluster database.

[0145] Exemplarily, the allocation selection of the ECG management node may be performed by randomly selecting from among the storage nodes in the cloud storage cluster.

[0146] 2. ECG lock.

[0147] For example, Figure 4 As shown, the complete process of ECG locking may include: ECG locking and ECG unlocking.

[0148] 2.1、ECG lock.

[0149] Exemplarily, when the storage node receives a data update request and determines the ECG corresponding to the data to be updated (which may be called the target ECG), it may send the data update request to the management node of the target ECG (which may be called the target ECG management node).

[0150] When receiving a data update request for the managed ECG, the target ECG management node may query the locking information in the memory to determine the locking status of the data to be updated.

[0151] When the locking state of the data to be updated is unlocked, the target ECG management node may lock the data to be updated.

[0152] Exemplarily, the data update request may include a locking strategy, and the locking strategy may include locking the entire ECG or locking a portion of the stripe.

[0153] For example, if the data to be updated covers three stripes of the target ECG, the three stripes may be locked.

[0154] Exemplarily, a locking strategy of locking the entire ECG is adopted by default.

[0155] For example, when the client requesting the lock has not been updated, the storage node can update the lock information (including the node that obtained the lock, time, lock version number, etc.) to the cluster database when it receives the lock request from the client for the first time; when it is not the first time to receive the lock request from the client, the lock information can be updated in the memory.

[0156] For example, to avoid the loss of locking information, the storage node may update the locking information recorded in the memory to the cluster database at regular intervals (eg, every 30 seconds).

[0157] For example, for a stripe locking strategy, for an ECG, a newly added stripe lock can be managed in the ECG management node memory to avoid performance loss caused by operating the cluster database.

[0158] For example, suppose an ECG includes 10 stripes, and client 1 requests to lock stripes 1 to 5. The relevant locking information is recorded in the memory and will also be recorded in the cluster database; if client 1 then requests to lock stripe 6, the management node only needs to update it in the memory and does not need to update the cluster database.

[0159] 2.2. ECG unlock.

[0160] For example, when the data update is completed, the lock release interface can be explicitly called, or the lock can be automatically released when a preset lock timeout period (such as 2 minutes) is reached.

[0161] The ECG management node may update the locking status of the corresponding data to be updated from locked to unlocked.

[0162] The above describes the method provided by this application. The following describes the device provided by this application:

[0163] See Figure 5 , is a structural diagram of a data processing device provided in an embodiment of the present application, wherein the data processing device can be deployed in a storage node in a fully symmetric cloud storage architecture, such as Figure 5 As shown, the data processing device may include:

[0164] The determining unit 510 is configured to determine the target file corresponding to the data to be updated and the target data block group in the target file corresponding to the data to be updated when receiving the data update request;

[0165] The determining unit 510 is further configured to determine the target data block group management node according to the information of the target data block group management node when metadata information of the target data block group and information of the target data block group management node corresponding to the target data block group are queried from the cluster database;

[0166] The determining unit 510 is further configured to determine the locking state corresponding to the data to be updated according to the locking information recorded in the memory when the storage node serves as the target data block group management node;

[0167] The processing unit 520 is configured to update the lock state corresponding to the data to be updated to locked when the lock state corresponding to the data to be updated is unlocked, and perform corresponding data update processing according to the data update request;

[0168] The processing unit 520 is further configured to refuse to execute the corresponding data update processing according to the data update request when the lock status corresponding to the data to be updated is locked.

[0169] In some embodiments, the processing unit 520 is further configured to send a data update request to the target data block group management node when the current storage node is not the target data block group management node, so that the target data block group management node determines the locking status corresponding to the data to be updated based on the locking information recorded in the memory; when the locking status corresponding to the data to be updated is unlocked, the locking status corresponding to the data to be updated is updated to locked, and the corresponding data update processing is performed according to the data update request; when the locking status corresponding to the data to be updated is locked, the corresponding data update processing is refused to be performed according to the data update request.

[0170] In some embodiments, the determining unit 510 is further configured to determine the target file management node based on the target file management node information when metadata information of the target data block group is not found from the cluster database, but metadata information of the target file and information of the target file management node corresponding to the target file are found;

[0171] The processing unit 520 is further configured to, when the current storage node serves as the target file management node, create a target data block group in the target file, assign a target data block group management node to the target data block group, and store metadata information of the target data block group and information of the target data block group management node in the cluster database; and, when the assigned target data block group management node is not the current storage node, send a data update request to the assigned target data block group management node.

[0172] In some embodiments, the processing unit 520 is further configured to create a target file, assign a target file management node to the target file, and store the metadata information of the target file and the information of the target file management node in the cluster database when the metadata information of the target file is not queried from the cluster database and the current storage node is the cluster master node; wherein the cluster master node is dynamically determined by the storage nodes in the cluster; and when the assigned target file management node is not the current storage node, send a data update request to the assigned target file management node.

[0173] In some embodiments, the data update request includes a locking strategy, which includes locking the entire data block group or stripe locking; a data block group includes multiple stripes;

[0174] The lock status corresponding to the data to be updated includes:

[0175] In the case where the locking strategy is to lock the entire data block group, if the locking state of the target data block group is locked, the locking state corresponding to the data to be updated is determined to be locked;

[0176] For the case where the locking strategy is stripe locking, when the locking status of at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking status corresponding to the data to be updated is determined to be locked; when the locking status of the stripes corresponding to the data to be updated in the target data block group are all unlocked, the locking status corresponding to the data to be updated is determined to be unlocked.

[0177] In some embodiments, when the data update request does not include a locking strategy, the locking strategy is determined to be locking the entire data block group.

[0178] In some embodiments, the processing unit 520 is further configured to update the locking state corresponding to the data to be updated to unlocked when the corresponding data update processing is completed according to the data update request or when a preset locking timeout period is reached.

[0179] See Figure 6 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 601, a communication interface 602, a memory 603, and a communication bus 604. The processor 601, the communication interface 602, and the memory 603 communicate with each other via the communication bus 604. The memory 603 stores a computer program; the processor 601 can execute the data processing method described above by executing the program stored in the memory 603.

[0180] The memory 603 mentioned herein may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the memory 603 may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0181] The present application also provides a non-transitory machine-readable storage medium storing a computer program, such as Figure 6 The computer program may be stored in the memory 603. Figure 6 The processor 601 in the electronic device shown executes to implement the data processing method described above.

[0182] The present application also provides a computer program stored in a non-transitory machine-readable storage medium, such as Figure 6 The memory 603 in the memory, and when the processor executes the computer program, it prompts the processor 601 to execute the data processing method described above.

Claims

1. A data processing method, characterized in that: Applied to a storage node in a fully symmetric cloud storage architecture, the method comprises: In the case of receiving a data update request, determining a target file corresponding to the data to be updated, and a target data block group in the target file corresponding to the data to be updated; When metadata information of the target data block group and information of a target data block group management node corresponding to the target data block group are queried from a cluster database, determining the target data block group management node according to the information of the target data block group management node; In a case where the storage node serves as the target data block group management node, determining a locking state corresponding to the data to be updated according to the locking information recorded in the memory; When the lock state corresponding to the data to be updated is unlocked, the lock state corresponding to the data to be updated is updated to locked, and corresponding data update processing is performed according to the data update request; When the lock state corresponding to the data to be updated is locked, the corresponding data update process is refused to be executed according to the data update request.

2. The method according to claim 1, characterized in that After determining the target data block group management node according to the information of the target data block group management node, the method further includes: In the case that the storage node is not the target data block group management node, the data update request is sent to the target data block group management node, so that the target data block group management node determines the locking status corresponding to the data to be updated based on the locking information recorded in the memory; in the case that the locking status corresponding to the data to be updated is unlocked, the locking status corresponding to the data to be updated is updated to locked, and the corresponding data update processing is performed according to the data update request; in the case that the locking status corresponding to the data to be updated is locked, the corresponding data update processing is refused to be performed according to the data update request.

3. The method according to claim 1, characterized in that The method further comprises: If metadata information of the target data block group is not found from the cluster database, but metadata information of the target file and information of the target file management node corresponding to the target file are found, determining the target file management node according to the information of the target file management node; In the case where the storage node serves as the target file management node, the target data block group is created in the target file, a target data block group management node is allocated to the target data block group, and metadata information of the target data block group and information of the target data block group management node are stored in the cluster database; In a case where the allocated target data block group management node is not the storage node, the data update request is sent to the allocated target data block group management node.

4. The method according to claim 1, wherein The method further comprises: When the metadata information of the target file is not found from the cluster database and the storage node is the cluster master node, the target file is created, a target file management node is assigned to the target file, and the metadata information of the target file and the information of the target file management node are stored in the cluster database; wherein the cluster master node is determined by dynamic negotiation among the storage nodes in the cluster; and when the assigned target file management node is not the storage node, the data update request is sent to the assigned target file management node.

5. The method according to claim 1, wherein The data update request includes a locking strategy, which includes locking the entire data block group or stripe locking; a data block group includes multiple stripes; The lock status corresponding to the data to be updated includes: In the case where the locking strategy is to lock the entire data block group, if the locking state of the target data block group is locked, determining that the locking state corresponding to the data to be updated is locked; For the case where the locking strategy is stripe locking, when the locking status of at least one stripe corresponding to the data to be updated in the target data block group is locked, the locking status corresponding to the data to be updated is determined to be locked; when the locking status of the stripes corresponding to the data to be updated in the target data block group are all unlocked, the locking status corresponding to the data to be updated is determined to be unlocked.

6. The method according to claim 5, characterized in that In a case where the data update request does not include a locking strategy, it is determined that the locking strategy is to lock the entire data block group.

7. The method according to claim 1, characterized in that After the locking state corresponding to the data to be updated is updated to locked, the method further includes: When the corresponding data update processing is completed according to the data update request, or when a preset lock timeout period is reached, the lock state corresponding to the data to be updated is updated to unlocked.

8. A data processing device, characterized in that: A storage node deployed in a fully symmetric cloud storage architecture, the device comprising: a determining unit configured to, upon receiving a data update request, determine a target file corresponding to the data to be updated and a target data block group in the target file corresponding to the data to be updated; The determining unit is further configured to determine the target data block group management node according to the information of the target data block group management node when metadata information of the target data block group and information of the target data block group management node corresponding to the target data block group are queried from the cluster database; The determining unit is further configured to determine, when the storage node serves as the target data block group management node, a locking state corresponding to the data to be updated based on the locking information recorded in the memory; a processing unit configured to update the locking state corresponding to the data to be updated to locked when the locking state corresponding to the data to be updated is unlocked, and perform corresponding data update processing according to the data update request; The processing unit is further configured to refuse to execute the corresponding data update processing according to the data update request when the locking state corresponding to the data to be updated is locked.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images