Data processing method and device, equipment and storage medium

Abstract

The application provides a data processing method, device and equipment and a storage medium. The method comprises the following steps: determining a target file corresponding to to-be-updated data and a target data block group; inquiring metadata information of the target data block group and information of a target data block group management node, and in the case that the current storage node is the target data block group management node, determining a locking state corresponding to the to-be-updated data according to locking information recorded in the memory; in the case that the locking state is not locked, updating the locking state to locked, and performing corresponding data update processing according to a data update request; in the case that the locking state is locked, refusing to perform corresponding data update processing according to the data update request. The method can improve the data processing performance while ensuring data consistency.

Description

Technical Field

[0001] This application relates to the field of cloud storage technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] With social development, security has become an increasingly important concern for people, and the widespread use of security monitoring has provided a certain degree of protection for people's safe lives.

[0003] Security monitoring systems contain a large amount of video data, which is currently typically stored in the cloud.

[0004] To achieve high availability of video data storage clusters, a fully symmetrical architecture is typically used for storing video data, meaning that each node in the cluster system has an equal role and jointly manages and maintains metadata.

[0005] Ensuring data consistency in a fully symmetric cloud storage architecture has become a popular research topic. Summary of the Invention

[0006] In view of this, this application provides a data processing method, apparatus, device, and storage medium.

[0007] According to a first aspect of the embodiments of this application, a data processing method is provided, applied to a storage node in a fully symmetric cloud storage architecture, the method comprising:

[0008] Upon receiving a data update request, the target file corresponding to the data to be updated, and the target data block group in the target file corresponding to the data to be updated are determined.

[0009] If the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found from the cluster database, the target data block group management node is determined based on the information of the target data block group management node.

[0010] When the storage node acts as the management node for the target data block group, the locking status corresponding to the data to be updated is determined based on the locking information recorded in memory.

[0011] If the lock status corresponding to the data to be updated is unlocked, the lock status corresponding to the data to be updated is updated to locked, and the corresponding data update process is performed according to the data update request;

[0012] If the lock status corresponding to the data to be updated is locked, the corresponding data update process is refused to be executed according to the data update request.

[0013] According to a second aspect of the embodiments of this application, a data processing apparatus is provided, deployed on a storage node in a fully symmetric cloud storage architecture, the apparatus comprising:

[0014] The determining unit is configured to, upon receiving a data update request, determine the target file corresponding to the data to be updated, and the target data block group in the target file corresponding to the data to be updated;

[0015] The determining unit is further configured to determine the target data block group management node based on the information of the target data block group management node when the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found from the cluster database.

[0016] The determining unit is further configured to, when the storage node is the management node of the target data block group, determine the locking state corresponding to the data to be updated based on the locking information recorded in memory;

[0017] The processing unit is configured to update the locking status of the data to be updated to locked when the locking status of the data to be updated is unlocked, and to perform corresponding data update processing according to the data update request.

[0018] The processing unit is also configured to refuse to perform the corresponding data update processing based on the data update request if the locking state corresponding to the data to be updated is locked.

[0019] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0020] Memory, used to store computer programs;

[0021] The processor, when executing a program stored in memory, implements the method provided in the first aspect.

[0022] According to a fourth aspect of the embodiments of this application, a non-transitory computer-readable storage medium is provided, wherein a computer program is stored in the non-transitory computer-readable storage medium, and the computer program implements the method provided in the first aspect when executed by a processor.

[0023] The data processing method of this application embodiment allocates a data block group management node to each data block group in the file, and stores the metadata information of the data block group and the metadata information of the data block group management node in a cluster database. When any storage node in the cloud storage cluster receives a data update request, it can query the metadata information of the corresponding target data block group and the target data block group management node in the cluster database. If the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found in the cluster database, the target data block group management node is determined based on the information of the target data block group management node. When a storage node acts as the target data block group management node, it can determine the locking status of the data to be updated based on the locking information recorded in memory. If the locking status of the data to be updated is unlocked, the locking status is updated to locked, and the corresponding data update processing is executed according to the data update request. If the locking status of the data to be updated is locked, the corresponding data update processing is refused according to the data update request. By assigning a data block group management node to the data block group, and having the data block group management node record the locking information related to the data block group in memory, data processing performance is improved while ensuring data consistency. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of an EC block group provided in an embodiment of this application;

[0026] Figure 3 This is a flowchart illustrating an allocation manager provided in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of an ECG locking process provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.

[0031] 1. Database (DB): A database is a structured collection of related data stored on a computer's storage device. A database contains various components, including tables, views, stored procedures, records, fields, and indexes.

[0032] 2. Video stream: refers to the transmission of video data, which can be processed over a network as a stable and continuous stream.

[0033] 3. DSD (Data Storage Device): Represents a storage node, which is a module in an object storage system for reading and writing object data. DSD stores data on disk tracks and sectors, combines several tracks and sectors to form an object, and provides external access to the data through this object.

[0034] 4. MDS (Management Data System): The management node in the data storage system, which stores the index information of objects, including the object name, the specific location where the object data is stored, and the object's last modification time.

[0035] 5. Cluster Technology: A cluster is a group of independent computers interconnected by a high-speed network, forming a group and managed as a single system. When a client interacts with a cluster, the cluster acts like a single server. Cluster configuration is used to improve availability and scalability.

[0036] 6. Domain: Cloud storage can manage multiple storage nodes. Data in resource pools created within a domain will only be written to storage resources within that domain, thus achieving domain-based isolation of storage resources.

[0037] 7. Symmetric: Based on the metadata management method, cluster file systems can be divided into symmetric and asymmetric types. In a symmetric cluster file system, each node has an equal role and jointly manages and maintains metadata. Nodes synchronize information and perform operations such as mutual exclusion locks through a high-speed network. In an asymmetric cluster file system, one or more nodes are dedicated to managing metadata, and other nodes need to frequently communicate with the metadata node to obtain the latest metadata, such as directory listings, file attributes, etc.

[0038] 8. Erasure Coding (ECG): An EC redundancy group consisting of multiple blocks. For example, a block size of 8MB, an EC of 4+2, and a unit size of 1MB. Six blocks form an ECG, containing eight 4+2 stripes with a unit size of 1MB each. Each block is sequentially divided into eight units (u0, u1, to u7), and the ux values ​​of each block form a 4+2 stripe.

[0039] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0040] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0041] Please see Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. For example, this data processing method can be applied to storage nodes in a fully symmetric cloud storage architecture, such as... Figure 1 As shown, the data processing method may include the following steps:

[0042] Step S100: Upon receiving a data update request, determine the target file corresponding to the data to be updated, and the target data block group within the target file corresponding to the data to be updated.

[0043] In this embodiment of the application, the user can send a data update request to the cloud storage cluster through the client, requesting to update the data in a specific data block group in a specific file.

[0044] For example, data updates include initial data writing or updates to data that has already been written.

[0045] When a storage node in a cloud storage cluster receives a data update request, it can determine the file corresponding to the data to be updated (which can be called the target file) and the group of data blocks in the target file (which can be called the target data block group).

[0046] For example, a file may include multiple data block groups, such as EC block groups, with different data block groups corresponding to different offsets in the file.

[0047] Storage nodes can determine the data block group corresponding to the data to be updated based on the offset of the data to be updated in the file.

[0048] Step S110: After querying the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group from the cluster database, determine the target data block group management node based on the information of the target data block group management node.

[0049] In this embodiment of the application, in order to improve data processing performance, a file management node can be assigned to the files created in the cloud storage cluster, and a data block group management node can be assigned to the data block groups created in the files. The data block group management node is used to process the data updates of the corresponding data block groups.

[0050] For example, file management nodes can be assigned by the master node of a cloud storage cluster.

[0051] For any given file, the data block group management node for each data block group within that file can be assigned by the file management node for that file.

[0052] For example, to avoid data block group management node allocation conflicts and ensure the consistency of data block group management node allocation, the file management node can allocate data block group management nodes through memory mutual exclusion. That is, for a data block group, when a certain thread of the file management node is allocating a data block group management node for that data block group, it can record the locking information for that data block group in memory to prevent other threads of the file management node from allocating different data block group management nodes for the same data block group.

[0053] Similarly, when the cluster master node allocates file management nodes to files in the cloud storage cluster, memory mutual exclusion can be used to ensure the consistency of file management node allocation.

[0054] For example, the master node of a cloud storage cluster is determined dynamically through negotiation among the storage nodes in the cloud storage cluster.

[0055] For example, the storage node with the largest IP address in the cloud storage cluster can be designated as the master node; in the event of a failure of the current master node, the storage node with the largest IP address among the remaining available storage nodes can be automatically designated as the new master node.

[0056] For example, if a file management node creates a data block group and assigns a data block group management node to the data block group, the metadata information of the data block group and the information of the corresponding data block group management node (such as the identification information of the data block management node, for example, IP address) can be stored in the cluster database.

[0057] For example, the metadata information of a data block group includes some or all of the information such as the block group ID (identifier) ​​and the location of the data block on the disk.

[0058] For example, for an EC block group, the metadata information of the data block group may also include the number of data blocks and check blocks in the data block group.

[0059] Accordingly, once the storage node has determined the target data block group corresponding to the data to be updated, it can query the metadata information of the target data block group in the cluster database, as well as the data block group management node (which can be called the target data block group management node) corresponding to the target data block group.

[0060] If the target data block group's metadata information and the target data block group's management node information are found in the cluster database, that is, if the target data block group and the target data block group's management node already exist, then the target data block group's management node can be determined based on the information of the target data block group's management node.

[0061] Step S120: When the storage node is the target data block group management node, determine the locking status of the data to be updated based on the locking information recorded in memory.

[0062] In this embodiment of the application, in order to improve data processing performance while ensuring data consistency, the data block group management node can record the relevant locking information of the managed data block group in memory (the memory of this management node).

[0063] Accordingly, when the target data block group management node receives a data update request, it can determine the locking status of the data to be updated based on the locking information recorded in memory.

[0064] For example, the locking status of the data to be updated can include locked or unlocked.

[0065] For example, if the lock status of the data to be updated is "locked", it means that the data is currently being updated; if the lock status of the data to be updated is "unlocked", it means that the data is not currently being updated.

[0066] It should be noted that, in this embodiment of the application, if the storage node (the storage node that receives the data update request in step S100) is not the target data block group management node, the storage node may send the data update request to the determined target data block group management node.

[0067] For example, the specific processing flow when the target data block group management node receives a data update request is similar to the processing flow when this storage node acts as the target data block group management node (see the relevant descriptions in steps 120 to 140), and will not be repeated here in this embodiment of the application.

[0068] Step S130: If the lock status of the data to be updated is unlocked, update the lock status of the data to be updated to locked, and perform the corresponding data update processing according to the data update request.

[0069] Step S140: If the lock status of the data to be updated is locked, refuse to perform the corresponding data update processing based on the data update request.

[0070] In this embodiment of the application, if the target data block group management node determines that the locking status of the data to be updated is unlocked, that is, the corresponding data is not currently in an update state, the target data block group management node can update the locking status of the data to be updated to locked, and perform the corresponding data update processing according to the data update request.

[0071] If the target data block group management node determines that the lock status of the data to be updated is locked, that is, the corresponding data is currently in an update state, in order to avoid data update conflicts and ensure data consistency, the target data block group management node may refuse to perform the corresponding data update processing based on the data update request.

[0072] It can be seen that, in Figure 1 In the illustrated method flow, a data block group management node is assigned to each data block group in the file, and the metadata information of the data block group and the metadata information of the data block group management node are stored in the cluster database. When any storage node in the cloud storage cluster receives a data update request, it can query the corresponding target data block group and the metadata information of the target data block group management node in the cluster database. If the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found in the cluster database, the target data block group management node is determined based on the information of the target data block group management node. In this storage node... When a node acts as the management node for the target data block group, it can determine the locking status of the data to be updated based on the locking information recorded in memory. If the locking status of the data to be updated is unlocked, the locking status is updated to locked, and the corresponding data update processing is executed according to the data update request. If the locking status of the data to be updated is locked, the corresponding data update processing is refused according to the data update request. By assigning a data block group management node to the data block group, and having the data block group management node record the locking information related to the data block group in memory, data processing performance is improved while ensuring data consistency.

[0073] In some embodiments, the data processing method provided in this application may further include:

[0074] If the metadata information of the target data block group is not found in the cluster database, but the metadata information of the target file and the information of the target file management node corresponding to the target file are found, the target file management node is determined based on the information of the target file management node.

[0075] When this storage node acts as the target file management node, a target data block group is created in the target file, a target data block group management node is assigned to the target data block group, and the metadata information of the target data block group and the information of the target data block group management node are stored in the cluster database.

[0076] If the target data block group management node is not this storage node, the data update request will be sent to the target data block group management node.

[0077] For example, when the cluster master node creates a file and assigns a file management node, it can store the file's metadata information and the corresponding file management node information (such as the file management node's identification information, for example, its IP address) in the cluster database.

[0078] If a storage node cannot find the metadata information of the target data block group in the cluster database, it can further query the metadata information of the target file and the file management node corresponding to the target file (which can be called the target file management node) in the cluster database.

[0079] If the target file's metadata information and the target file management node's information are found in the cluster database, meaning the target file already exists and the target file management node also exists, then the target file management node can be determined based on the target file management node's information.

[0080] When this storage node is the target file management node, it can create target data block groups in the target file and assign a target data block group management node to the target data block group.

[0081] For example, the target file management node can be selected randomly from among the storage nodes in the cloud storage cluster, or the storage node that initially receives the data update request can be selected as the target data block group management node.

[0082] It should be noted that, in order to ensure the availability of the data block group management node, offline nodes need to be excluded during the selection process. In addition, to avoid excessive load on the data block group management node, storage nodes with CPU utilization exceeding a preset utilization threshold or memory utilization exceeding a preset utilization threshold can also be excluded during the selection process.

[0083] For example, the target file management node can store the metadata information of the target data block group and the information of the target data block group management node in the cluster database.

[0084] For example, if the assigned target data block group management node is not this storage node (the storage node that is the target file management node), the target file management node can send the data update request to the target data block group management node.

[0085] The processing flow after the target data block group management node receives the data update request can be found in the relevant description in the above embodiments, and will not be repeated in this application embodiment.

[0086] It should be noted that when the management node of the allocated target data block group is this storage node, the storage node can perform the relevant processing in the manner described in steps S120 to S140, which will not be repeated here in the embodiments of this application.

[0087] In some embodiments, the data processing method provided in this application may further include:

[0088] If the target file's metadata information is not found in the cluster database, and this storage node is the cluster master node, the target file is created, a target file management node is assigned to the target file, the target file's metadata information and the target file management node's information are stored in the cluster database, and a data update request is sent to the target file management node; the cluster master node is dynamically determined by negotiation among the storage nodes in the cluster.

[0089] If the target file management node assigned is not this storage node, the data update request will be sent to the assigned target file management node.

[0090] For example, if a storage node cannot find the metadata information of the target file in the cluster database, it can determine that the target file has not yet been created. In this case, the target file creation process needs to be performed.

[0091] When a storage node acts as the cluster master node, it can create target files and assign a target file management node to the target files.

[0092] For example, the cluster master node can randomly select the target file management node from among the storage nodes in the cloud storage cluster, or prioritize the storage node that initially receives the data update request as the target file management node.

[0093] It should be noted that, in order to ensure the availability of file management nodes, offline nodes need to be excluded during the selection process. In addition, to avoid excessive load on file management nodes, storage nodes with CPU utilization exceeding a preset utilization threshold or memory utilization exceeding a preset utilization threshold can also be excluded during the selection process.

[0094] For example, the cluster master node can store the metadata information of the target file and the information of the target file management node in the cluster database.

[0095] It should be noted that, in this embodiment of the application, if the storage node (i.e., the storage node that receives the data update request in step S100) is not the cluster master node, the data update request can be sent to the queried cluster master node.

[0096] The specific processing flow when the cluster master node receives a data update request sent by the storage node is similar to the processing flow when the storage node is the cluster master node (see the relevant description in the above embodiments), and will not be repeated here.

[0097] For example, if the assigned target file management node is not the storage node (the storage node that acts as the cluster master node), the cluster master node can send data update requests to the target file management node.

[0098] When the target file management node receives a data update request, it can perform target data block group creation processing. The specific processing flow can be found in the relevant description in the above embodiments, and will not be repeated in this application embodiment.

[0099] Once the target file management node has completed the creation of the target data block group and the allocation of the target data block group management node, it can send data update requests to the target data block group management node.

[0100] The processing flow after the target data block group management node receives the data update request can be found in the relevant description in the above embodiments, and will not be repeated in this application embodiment.

[0101] It should be noted that when the allocated target file management node is this storage node, the storage node can perform the creation of the target data block group and the allocation of the target data block group management node in the manner described in the above embodiments. For the specific implementation, please refer to the relevant description in the above embodiments. This application embodiment will not repeat it here.

[0102] In some embodiments, the data update request may include a locking strategy, which may include locking the entire data block group or stripe locking; a data block group includes multiple stripes;

[0103] The locking status corresponding to the data to be updated mentioned above can include:

[0104] When the locking strategy locks the entire data block group, if the target data block group is already locked, the locking status of the data to be updated is determined to be already locked.

[0105] For cases where the locking strategy is stripe locking, if at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking state of the data to be updated is determined to be locked; if all stripes in the target data block group corresponding to the data to be updated are unlocked, the locking state of the data to be updated is determined to be unlocked.

[0106] For example, to improve locking flexibility, instead of being limited to locking on a stripe-by-strip basis, it is possible to choose to lock the entire data block group as needed, or to lock one or more stripes within the data block group.

[0107] For example, a data block group may consist of multiple stripes.

[0108] For example, taking EC block groups as an example, such as Figure 2 As shown, suppose an EC block group consists of 6 data blocks, each data block is divided into 8 units (let's say u0 to u7), and the ux (x = 0, 1, ..., 7) of each block form a 4+2 stripe. For example, the u0 of each block forms a 4+2 stripe.

[0109] Correspondingly, when users initiate a data update request through the client, they can also select a locking strategy as needed and send the locking strategy along with the data update request to the storage nodes of the cloud storage cluster.

[0110] When a storage node receives a data update request, it can obtain the locking request included in the data update request.

[0111] For example, when the locking strategy locks the entire data block group, if the target data block group is already locked, the locking status of the data to be updated is determined to be already locked.

[0112] For cases where the locking strategy is stripe locking, if at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking state of the data to be updated is determined to be locked; if all stripes in the target data block group corresponding to the data to be updated are unlocked, the locking state of the data to be updated is determined to be unlocked.

[0113] In one example, if the locking strategy is not included in the data update request, the locking strategy is determined to lock the entire data block group.

[0114] For example, in order to optimize locking performance and improve locking efficiency, the default locking strategy can lock the entire data block group.

[0115] Accordingly, if the data update request does not include a locking strategy, the locking strategy is determined to lock the entire data block group.

[0116] In some embodiments, after updating the locking status of the data to be updated to locked, the process may further include:

[0117] Once the corresponding data update process is completed according to the data update request, or if the preset locking timeout period is reached, the locking status of the data to be updated will be updated to unlocked.

[0118] For example, to avoid deadlocks or lock leaks, unlocking can support active release as well as a timeout release mechanism.

[0119] Accordingly, the data block group management node can proactively perform unlocking processing after completing the corresponding data update processing based on the data update request.

[0120] For example, the data block group management node can explicitly call the lock release interface to release the lock and update the locking status of the corresponding data from locked to unlocked.

[0121] In addition, a lock timeout period can be preset, such as 2 minutes. When the data block group management node determines that the preset lock timeout period has been reached, the lock release process must be performed regardless of whether the data update process is completed, and the lock status of the corresponding data will be updated from locked to unlocked.

[0122] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0123] In this embodiment of the application, in order to improve locking efficiency while ensuring data consistency, the concept of file management node / data block group (taking EC block group (ECG) as an example) management node is proposed, and ECG-level locking granularity is provided.

[0124] Each file and ECG is assigned a corresponding management node. Actions such as creating ECGs, locking and updating ECGs under a given file are all handled by the corresponding management node. The management node performs concurrent mutual exclusion operations in memory, ensuring locking consistency and avoiding the use of distributed locks. Every node in the cluster can be elected as a management node, eliminating single-node performance bottlenecks and single-node failure issues.

[0125] In addition, the program acquires locks at the default ECG level, improving locking efficiency. The system's read / write update strategy maximizes the guarantee that updates to the same ECG are initiated by the same client, reducing stripe-level locking.

[0126] The specific processing procedure is explained below.

[0127] In this embodiment, the data processing flow may include two parts: allocation manager and ECG locking.

[0128] I. Assignment of managers.

[0129] For example, such as Figure 3 As shown, the process of assigning a manager may include: the cluster master node assigning a file management node; and the file management node assigning an ECG management node.

[0130] 1.1 The cluster master node is assigned a file management node.

[0131] For example, the master node of a cloud storage cluster can be determined through dynamic negotiation among the storage nodes.

[0132] For any file, during the creation process, the cluster master node can assign a file management node to that file.

[0133] For example, to ensure consistency in file creation, the cluster master node can use memory mutual exclusion to create files, avoiding the concurrent creation of two copies of the metadata index for a single file.

[0134] For example, to ensure consistency in the allocation of file management nodes, the cluster master node can allocate file management nodes using a memory mutual exclusion method.

[0135] For example, the cluster master node can select the file management node by randomly selecting it from among the storage nodes in the cloud storage cluster.

[0136] For example, when the cluster master node completes file creation and file management node allocation, it can record file metadata information and file management node information in the cluster database.

[0137] 1.2. Assign ECG management nodes to file management nodes.

[0138] For example, a file may include multiple ECGs, and updating the data within the file requires locating the corresponding ECG based on its offset within the file.

[0139] The number of ECGs included in a file can be configured according to requirements; for example, a single file can be limited to a maximum of 50 ECGs.

[0140] To ensure data consistency during updates, the ECG needs to be locked before updating the corresponding ECG data.

[0141] To avoid the ECG locking logic involving distributed locks, a management node can be assigned to the ECG.

[0142] For example, an ECG management node can be assigned to each ECG by the file management node.

[0143] For example, file management nodes can allocate ECG management nodes through memory mutual exclusion to ensure the consistency of ECG management node allocation.

[0144] In addition, the file management node can record the information of the ECG management node and the ECG metadata information together in the cluster database.

[0145] For example, the allocation of ECG management nodes can be done by randomly selecting them from among the storage nodes in the cloud storage cluster.

[0146] 2. Lock the ECG.

[0147] For example, such as Figure 4 As shown, the complete ECG locking process can include ECG locking and ECG unlocking.

[0148] 2.1. Lock ECG.

[0149] For example, when a storage node receives a data update request and determines the ECG (which can be called the target ECG) corresponding to the data to be updated, it can send the data update request to the management node of the target ECG (which can be called the target ECG management node).

[0150] When a target ECG management node receives a data update request for the ECG it manages, it can query the locking information in memory to determine the locking status of the data to be updated.

[0151] If the data to be updated is in an unlocked state, the target ECG management node can lock the data to be updated.

[0152] For example, a data update request may include a locking strategy, which may include locking the entire ECG or locking a portion of the stripes.

[0153] For example, if the data to be updated covers three stripes of the target ECG, then those three stripes can be locked.

[0154] For example, the default locking strategy is to lock the entire ECG.

[0155] For example, if the client requesting the lock has not updated, the storage node can send the locking information (including the node that acquired the lock, the time, the lock version number, etc.) to the cluster database when it first receives the locking request from the client; if it receives the locking request from the client again, it can update the locking information in memory.

[0156] For example, to avoid losing locking information, storage nodes can periodically (e.g., every 30 seconds) update the locking information recorded in memory to the cluster database.

[0157] For example, for a striped locking strategy, the newly added striped locking for an ECG can be managed in memory on the ECG management node, avoiding performance loss caused by operating the cluster database.

[0158] For example, suppose an ECG consists of 10 stripes. Client 1 requests to lock stripes 1 to 5. The relevant locking information is recorded in memory and also in the cluster database. If client 1 then requests to lock stripe 6, the management node only needs to update the information in memory and does not need to update the cluster database.

[0159] 2.2 ECG unlocking.

[0160] For example, when the data update is complete, the lock release interface can be explicitly called, or the lock can be automatically released when the preset lock timeout time (such as 2 minutes) is reached.

[0161] The ECG management node can update the locking status of the corresponding data to be updated from locked to unlocked.

[0162] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0163] Please see Figure 5 This is a schematic diagram of a data processing device provided in an embodiment of this application. The data processing device can be deployed on a storage node in a fully symmetrical cloud storage architecture, such as... Figure 5 As shown, the data processing apparatus may include:

[0164] The determining unit 510 is configured to, upon receiving a data update request, determine the target file corresponding to the data to be updated, and the target data block group in the target file corresponding to the data to be updated;

[0165] The determining unit 510 is also configured to determine the target data block group management node based on the information of the target data block group management node when the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found from the cluster database.

[0166] The determining unit 510 is also configured to determine the locking status of the data to be updated based on the locking information recorded in memory when the storage node is the target data block group management node.

[0167] The processing unit 520 is configured to update the locking status of the data to be updated to locked when the locking status of the data to be updated is unlocked, and to perform the corresponding data update processing according to the data update request.

[0168] The processing unit 520 is also configured to refuse to perform the corresponding data update processing based on the data update request if the lock status of the data to be updated is locked.

[0169] In some embodiments, the processing unit 520 is further configured to, when the storage node is not the target data block group management node, send a data update request to the target data block group management node, so that the target data block group management node determines the locking status corresponding to the data to be updated based on the locking information recorded in memory; if the locking status corresponding to the data to be updated is unlocked, update the locking status corresponding to the data to be updated to locked, and perform the corresponding data update processing according to the data update request; if the locking status corresponding to the data to be updated is locked, refuse to perform the corresponding data update processing according to the data update request.

[0170] In some embodiments, the determining unit 510 is further configured to determine the target file management node based on the information of the target file management node when the metadata information of the target data block group is not found from the cluster database, but the metadata information of the target file and the information of the target file management node corresponding to the target file are found.

[0171] The processing unit 520 is also configured to, when the storage node is the target file management node, create a target data block group in the target file, assign a target data block group management node to the target data block group, and store the metadata information of the target data block group and the information of the target data block group management node in the cluster database; when the assigned target data block group management node is not the storage node, send the data update request to the assigned target data block group management node.

[0172] In some embodiments, the processing unit 520 is further configured to, when the metadata information of the target file is not found in the cluster database and the storage node is the cluster master node, create the target file, assign a target file management node to the target file, and store the metadata information of the target file and the information of the target file management node in the cluster database; wherein, the cluster master node is dynamically determined by the storage nodes in the cluster through negotiation; and when the assigned target file management node is not the storage node, send a data update request to the assigned target file management node.

[0173] In some embodiments, the data update request includes a locking strategy, which includes locking the entire data block group or stripe locking; a data block group includes multiple stripes;

[0174] The lock states corresponding to the data to be updated include:

[0175] When the locking strategy locks the entire data block group, if the target data block group is already locked, the locking status of the data to be updated is determined to be already locked.

[0176] For cases where the locking strategy is stripe locking, if at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking state of the data to be updated is determined to be locked; if all stripes in the target data block group corresponding to the data to be updated are unlocked, the locking state of the data to be updated is determined to be unlocked.

[0177] In some embodiments, if a locking strategy is not included in the data update request, the locking strategy is determined to lock the entire data block group.

[0178] In some embodiments, the processing unit 520 is further configured to update the locked state of the data to be updated to unlocked when the corresponding data update processing is completed according to the data update request, or when a preset locking timeout period is reached.

[0179] Please see Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 601, a communication interface 602, a memory 603, and a communication bus 604. The processor 601, communication interface 602, and memory 603 communicate with each other via the communication bus 604. The memory 603 stores a computer program; the processor 601 can execute the data processing method described above by executing the program stored in the memory 603.

[0180] The memory 603 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the memory 603 can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof.

[0181] This application also provides a non-transitory machine-readable storage medium storing a computer program, such as... Figure 6 The memory 603 in the computer program can be used by Figure 6 The processor 601 in the illustrated electronic device executes to implement the data processing method described above.

[0182] This application also provides a computer program stored on a non-transitory machine-readable storage medium, such as... Figure 6 The memory 603 in the memory, and when the processor executes the computer program, it causes the processor 601 to perform the data processing method described above.

Claims

1. A data processing method, characterized in that, The method, applied to storage nodes in a fully symmetric cloud storage architecture, includes: Upon receiving a data update request, the target file corresponding to the data to be updated, and the target data block group in the target file corresponding to the data to be updated are determined. If the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found from the cluster database, the target data block group management node is determined based on the information of the target data block group management node. When the storage node acts as the management node for the target data block group, the locking status corresponding to the data to be updated is determined based on the locking information recorded in memory. If the lock status corresponding to the data to be updated is unlocked, the lock status corresponding to the data to be updated is updated to locked, and the corresponding data update process is performed according to the data update request; If the lock status corresponding to the data to be updated is locked, the corresponding data update processing will not be performed according to the data update request. If the storage node is not the target data block group management node, the data update request is sent to the target data block group management node so that the target data block group management node determines the locking status of the data to be updated based on the locking information recorded in memory; if the locking status of the data to be updated is unlocked, the locking status of the data to be updated is updated to locked, and the corresponding data update process is executed according to the data update request; if the locking status of the data to be updated is locked, the execution of the corresponding data update process according to the data update request is refused. The data update request includes a locking strategy, which may include locking the entire data block group or locking the stripes; a data block group may include multiple stripes. The locking state corresponding to the data to be updated includes: When the locking strategy locks the entire data block group, if the locking status of the target data block group is locked, the locking status of the data to be updated is determined to be locked. When the locking strategy is stripe locking, if at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking state corresponding to the data to be updated is determined to be locked; if all stripes in the target data block group corresponding to the data to be updated are unlocked, the locking state corresponding to the data to be updated is determined to be unlocked.

2. The method according to claim 1, characterized in that, The method further includes: If the metadata information of the target data block group is not found in the cluster database, but the metadata information of the target file and the information of the target file management node corresponding to the target file are found, the target file management node is determined based on the information of the target file management node. When the storage node acts as the target file management node, the target data block group is created in the target file, a target data block group management node is assigned to the target data block group, and the metadata information of the target data block group and the information of the target data block group management node are stored in the cluster database. If the assigned target data block group management node is not the storage node, the data update request will be sent to the assigned target data block group management node.

3. The method according to claim 1, characterized in that, The method further includes: If the target file's metadata information is not found in the cluster database, and the storage node is the cluster master node, the target file is created, a target file management node is assigned to the target file, and the target file's metadata information and the target file management node's information are stored in the cluster database; wherein, the cluster master node is dynamically determined by negotiation among the storage nodes in the cluster; if the assigned target file management node is not the storage node, the data update request is sent to the assigned target file management node.

4. The method according to claim 1, characterized in that, If the data update request does not include a locking strategy, the locking strategy is determined to be locking the entire data block group.

5. The method according to claim 1, characterized in that, After updating the locking status of the data to be updated to "locked", the process further includes: Upon completion of the corresponding data update process based on the data update request, or upon reaching the preset locking timeout period, the locking status of the data to be updated is updated to unlocked.

6. A data processing apparatus, characterized in that, A storage node deployed in a fully symmetric cloud storage architecture, the device comprising: The determining unit is configured to, upon receiving a data update request, determine the target file corresponding to the data to be updated, and the target data block group in the target file corresponding to the data to be updated; The determining unit is further configured to determine the target data block group management node based on the information of the target data block group management node when the metadata information of the target data block group and the information of the target data block group management node corresponding to the target data block group are found from the cluster database. The determining unit is further configured to, when the storage node is the management node of the target data block group, determine the locking state corresponding to the data to be updated based on the locking information recorded in memory; The processing unit is configured to update the locking status of the data to be updated to locked when the locking status of the data to be updated is unlocked, and to perform corresponding data update processing according to the data update request. The processing unit is also configured to refuse to perform the corresponding data update processing according to the data update request if the locking state corresponding to the data to be updated is locked. The processing unit is further configured to, when the storage node is not the target data block group management node, send the data update request to the target data block group management node, so that the target data block group management node determines the locking status corresponding to the data to be updated based on the locking information recorded in memory; when the locking status corresponding to the data to be updated is unlocked, update the locking status corresponding to the data to be updated to locked, and perform the corresponding data update processing according to the data update request; when the locking status corresponding to the data to be updated is locked, refuse to perform the corresponding data update processing according to the data update request. The data update request includes a locking strategy, which may include locking the entire data block group or locking the stripes; a data block group may include multiple stripes. The locking state corresponding to the data to be updated includes: When the locking strategy locks the entire data block group, if the locking status of the target data block group is locked, the locking status of the data to be updated is determined to be locked. When the locking strategy is stripe locking, if at least one stripe in the target data block group corresponding to the data to be updated is locked, the locking state corresponding to the data to be updated is determined to be locked; if all stripes in the target data block group corresponding to the data to be updated are unlocked, the locking state corresponding to the data to be updated is determined to be unlocked.

7. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images