Data access control method and apparatus for distributed storage system

Abstract

Embodiments of the present application provide a data access control method and apparatus for a distributed storage system, capable of optimizing the overall performance of the distributed storage system. The method comprises: acquiring at least one data access request for a target data object, and generating corresponding access record data for the data access request; on the basis of the access record data, generating a corresponding data object copy for the target data object; determining result state information of the at least one data access request, and merging a plurality of data object copies on the basis of the result state information to determine access result data corresponding to the target data object; and updating the target data object on the basis of the access result data.

Description

A distributed storage system data access control method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on October 31, 2023, with application number 202311426714.4, entitled “A Method and Device for Data Access Control of a Distributed Storage System”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of distributed storage technology, and in particular to a method and device for controlling data access in a distributed storage system. Background Art

[0004] In distributed systems, operations often involve data on multiple Input / Output (IO) nodes. These operations require that all data be modified completely or not at all, with no intermediate states allowed. This is commonly referred to as transaction consistency. Currently, the mainstream approach to achieving consistency is distributed locking. When multiple clients operate on the same data object, lock conflicts can occur. A later client must wait for the earlier client to release the distributed lock before it can successfully acquire the lock and perform the corresponding data IO operation.

[0005] With the continuous advancement of storage technology, the latency of storage media has increased from milliseconds to microseconds, such as the transition from hard disk drives (HDDs) to storage-class memory (SCM). Distributed locking mechanisms can create bottlenecks in the overall latency performance of distributed systems. The current fastest network latency is 0.6 microseconds, with a request and a response taking a total of 1.2 microseconds. Applying and releasing both requests takes 2.4 microseconds, and this is just the network overhead, not including other processing overhead. This overhead was negligible in the HDD era, but in the SCM era, the excessive proportion of application and release time overhead associated with distributed locks has become prominent, negatively impacting the overall performance optimization of distributed systems.

[0006] Summary of the Invention

[0007] In one aspect, an embodiment of the present application provides a distributed storage system data access control method, comprising:

[0008] Obtain at least one data access request for a target data object, and generate corresponding access record data for each data access request;

[0009] Generate corresponding multiple data object copies for the target data object according to the access record data;

[0010] Determining result status information of at least one data access request, and merging multiple data object copies based on the result status information to determine access result data corresponding to the target data object; and

[0011] Update the target data object according to the access result data.

[0012] On the other hand, an embodiment of the present application further provides a distributed storage system data access control device, comprising:

[0013] An access record module, configured to obtain at least one data access request for a target data object and generate corresponding access record data for each data access request;

[0014] A copy generation module, used for generating corresponding multiple data object copies for the target data object according to the access record data;

[0015] a replica merging module, configured to determine result status information of at least one data access request, and merge multiple data object replicas based on the result status information to determine access result data corresponding to the target data object; and

[0016] And a data update module is used to update the target data object according to the access result data.

[0017] On the other hand, an embodiment of the present application also provides an electronic device for distributed storage system data access control, including: one or more processors; and a memory associated with the one or more processors, the memory being used to store computer-readable instructions, which implement the above-mentioned distributed storage system data access control method when read and executed by the one or more processors.

[0018] On the other hand, an embodiment of the present application further provides a non-transitory computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the distributed storage system data access control method as described above is implemented.

[0019] As can be seen from the above, the embodiments of the present application provide a distributed storage system data access control method, device, electronic device and storage medium, which receive data access requests for target data objects, generate access record data for the data access requests, and generate multiple data object copies for the target data object based on the access request data. The multiple data object copies are then merged based on the actual operation result status information to determine the access result data to update the data object. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The features and advantages of the present application will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present application in any way. In the accompanying drawings:

[0021] FIG1 is a schematic diagram showing the working principle of a distributed lock mechanism in a distributed storage system;

[0022] FIG2 shows a schematic diagram of a data access control method for a distributed storage system provided by one or more embodiments of the present application;

[0023] FIG3 shows a schematic diagram of a method for generating a data object copy for a data object in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0024] FIG4 shows a schematic diagram of a distributed storage system structure of a distributed storage system data access control method provided by one or more embodiments of the present application;

[0025] FIG5 shows a schematic diagram of a method for deduplicating data object distribution in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0026] FIG6 shows a schematic diagram of a method for merging data object copies in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0027] FIG7-a shows a schematic diagram of a merging process of data object copies in which a T1 data access request fails and a T2 data access request succeeds in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0028] FIG7-b is a schematic diagram showing a result of merging data object copies in which a data access request T1 succeeds and a data access request T2 fails in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0029] FIG8-a shows a schematic diagram of a merging process of data object copies in which T1 data access request succeeds and T2 data access request fails in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0030] FIG8-b is a schematic diagram showing a result of merging data object copies in which a data access request T1 succeeds and a data access request T2 fails in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0031] FIG9 is a schematic diagram showing a method for batch processing of access record data in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0032] FIG10 is a schematic diagram showing a method for batch processing of data object copies in a distributed storage system data access control method provided by one or more embodiments of the present application;

[0033] FIG11 shows a schematic structural diagram of a data access control device for a distributed storage system provided by one or more embodiments of the present application;

[0034] FIG12 shows a schematic structural diagram of a distributed storage system data access control electronic device provided by one or more embodiments of the present application;

[0035] FIG13 shows a schematic structural diagram of a non-transitory computer-readable storage medium provided by one or more embodiments of the present application. DETAILED DESCRIPTION

[0036] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0037] In distributed storage systems, operations often involve data on multiple I / O nodes, requiring that all data be modified completely or not at all, with no intermediate states allowed. This is commonly referred to as transaction consistency. Currently, the mainstream approach to achieving consistency is distributed locking. When multiple clients operate on the same data object, lock conflicts can occur.

[0038] As shown in Figure 1, if client A and client B modify the same object, client A first obtains a distributed lock from the metadata server before performing any I / O operations on the object. If client B also wants to operate on the object at this time, it will also issue the same distributed lock request to the metadata server. Upon determining a lock conflict, client B waits until client A releases the distributed lock before successfully re-acquiring the lock from the metadata server and performing the corresponding I / O operations. The existence of the distributed lock mechanism requires that the subsequent client must wait for the previous client to release the distributed lock before successfully acquiring the lock and performing the corresponding data I / O operations.

[0039] With the continuous advancement of storage technology, storage media latency has increased from milliseconds to microseconds, for example, from mechanical hard disks (HDDs) to storage-class memory (SCM). Distributed locking mechanisms can create bottlenecks in the overall latency performance of distributed systems. The current fastest network latency is 0.6 microseconds, with a request and a response taking a total of 1.2 microseconds. Applying and releasing both requests takes 2.4 microseconds, and this represents only network overhead, not including other processing overhead. This overhead was negligible in the HDD era, but in the SCM era, the excessive overhead associated with applying and releasing distributed locks has become prominent, negatively impacting the overall performance optimization of distributed systems.

[0040] To address the above issues, the present invention aims to propose a data access control method for a distributed storage system. This method generates multiple copies of data objects in response to data operation requests from different clients, and then determines the final data operation result through further judgment and merging. This approach replaces the distributed locking mechanism used in some related technologies, ensuring data consistency while avoiding the impact of distributed locking mechanisms on the performance optimization of distributed storage data access, thereby further optimizing system performance.

[0041] Based on the above objectives, on one hand, an embodiment of the present application provides a data access control method for a distributed storage system.

[0042] As shown in FIG2 , one or more embodiments of the present application provide a method for controlling data access to a distributed storage system. The method can be applied to a distributed storage system. The method includes:

[0043] S1: Obtain at least one data access request for a target data object, and generate corresponding access record data for the data access request.

[0044] In a distributed storage system, a data object refers to an object in object storage, a data structure, or a continuous storage space. Data objects can reside on the same physical machine, on different physical machines, or across different regions. In a distributed storage system, a single client can access different data objects, and multiple clients can access the same data object. In practical applications of distributed storage systems, multiple clients may simultaneously perform access operations on multiple data objects. In response to determining that the access operation is a read operation, content data can be retrieved from the corresponding data object in response to the data access request.

[0045] In a distributed storage system, one or more data access requests for a target data object may be obtained. If there are multiple data access requests for a target data object, the multiple data access requests may come from the same client or from multiple different clients.

[0046] After obtaining the data access request, access operation information that needs to be performed on the target data object can be extracted from the data access request, and access record data corresponding to the target data object can be generated based on the access operation information.

[0047] The generated access record data can be stored in a write-ahead log (WAL), which can be stored in millisecond-level SCM storage media. Storing access record data in SCM storage media can achieve low latency and fast response for data record transmission.

[0048] For each data access request corresponding to a target data object, an access record is generated. In practical applications, multiple data objects in a distributed storage system may receive multiple data access requests simultaneously. For multiple data objects, data access control operations based on data access requests can be executed synchronously and in parallel, improving the overall data access efficiency of the distributed storage system. For a specific target data object, data access operations based on the corresponding data access request are executed asynchronously to ensure the accuracy of data operation results.

[0049] S2: Generate a corresponding data object copy for the target data object based on the access record data.

[0050] At least one access record data item may be generated for at least one data access request corresponding to the target data object. In response to the at least one access record data item, multiple data object copies may be generated for the target data object.

[0051] The data object replica is used to represent a possible access operation result of an access operation on a target data object. The target data object and the corresponding data object replica can be set in a storage memory of the distributed storage system.

[0052] When generating a data object copy for a data object based on at least one access record data corresponding to a target data object, the data object copy can be generated in sequence based on at least one access record data in the order of request time of at least one data access request corresponding to at least one access record data.

[0053] When there is only one access record data corresponding to the data object, a data object copy can be directly generated for the data object according to the access operation information in the access record data.

[0054] When a data object has multiple (two or more) access records corresponding to the multiple access records, a sequence of request times for the multiple data access requests corresponding to the multiple access records is determined. A data object copy is then generated for the data object based on the access operation information in the corresponding access records, sequentially according to the sequence of request times for the multiple data access requests.

[0055] In some embodiments, based on the order of request time, the access record data corresponding to the first-ranked data access request is determined, and a corresponding data object copy is first generated for the data object based on the access record data. Then, a data object copy is generated for the access record data corresponding to the second-ranked data access request, and data object copies are generated for the access record data corresponding to subsequent data access requests in sequence until corresponding data object copies are generated for all access record data corresponding to the data object.

[0056] As shown in FIG3 , in a distributed storage system data access control method provided by one or more embodiments of the present application, multiple data object copies are generated sequentially for a target data object, including:

[0057] S201: Determine at least one data object copy corresponding to an access record data previous to the current access record data.

[0058] S202: Generate corresponding data object copies according to the access operation information in the current access record data for two different result states of the data access operation corresponding to the previous access record data, wherein the result states include data access operation success and data access operation failure.

[0059] In response to determining that the data access request time corresponding to the current access record data is ranked first, a data object copy can be generated based on the initial data content in the data object and the access operation information in the current access record data.

[0060] As shown in Figure 4, in a distributed storage system, taking the example of a client (T1, T2) issuing a data access request for five data objects (m1:o1, m1:o2, m2:o1, m2:o2, m3:o1) located in three storage devices (machine1, machine2, machine3, which can also be recorded as m1, m2, m3), the method of generating data copy objects is explained.

[0061] Two clients T1 and T2 issue multiple data write operation requests for five data objects m1:o1, m1:o2, m2:o1, m2:o2, and m3:o1. The access record data corresponding to the multiple data access requests can be expressed as follows:

[0062] m1:o1:v1; m1:o2:v1; m2:o1:v1; m3:o1:v1; m1:o2:v2; m2:o2:v1…

[0063] Here, m1:o1:v1 indicates that a write-modify operation with version v1 is performed on the data object o1 on the storage device m1.

[0064] For data object o2 on storage device m1, T1 first performs a write-modify operation on it with version v1, and then T2 performs a write-modify operation on it with version v2. The corresponding access records are m1:o2:v1 and m1:o2:v2.

[0065] In FIG4 , a solid-line frame represents a data object, and a dotted-line frame represents a data object copy.

[0066] For the access record data m1:o1:v1, a corresponding data object copy Object1_v1 is generated for the data object Object1_v0 of the storage device m1.

[0067] For the access record data m1:o2:v1 and m1:o2:v2, for the data object Object2_v0 of the storage device m2, first generate the corresponding data object copy Object2_v1 corresponding to the access record data m1:o2:v1, and then generate two data object copies corresponding to the location record data m1:o2:v2. The two data object copies are:

[0068] When the data access operation of T1 succeeds, Object2_v1+Object2_v2;

[0069] When the data access operation of T1 fails, Object2_v2.

[0070] The two data object copies generated by the two operations for the data object Object2_v0 on the storage device m2 are connected by a linked list (indicated by a dotted line in FIG4 ). The data object Object2_v0 and the corresponding multiple data object copies are connected by a bidirectional linked list (indicated by a solid line in FIG4 ).

[0071] In some embodiments, a local lock is further set in the memory corresponding to the target data object in the distributed storage system to maintain a bidirectional linked list between the data object and the corresponding multiple data object copies.

[0072] In this way, in response to at least one data access request for a target data object, a plurality of associated data object copies can be generated in the memory where the target data object is located.

[0073] In actual application scenarios, different data access requests for a target data object may perform the same access operation on the data object, and the corresponding data object copies may contain duplicate content. In this regard, as shown in FIG5 , in a distributed storage system data access control method provided by one or more embodiments of the present application, before merging multiple data object copies corresponding to the data object based on the result status information, the method further includes:

[0074] S301: Compare the contents of data object copies corresponding to two adjacent data access requests to determine whether the two adjacent data access requests perform the same access operation on the data object.

[0075] Two adjacent data access requests refer to data access requests that are adjacent in access time sequence. In some implementations, the contents of the data object replicas corresponding to the two adjacent data access requests can be compared using the following method: calculating a hash value for the data object replicas corresponding to the two adjacent data access requests to determine whether the hash values ​​of the data object replicas corresponding to the two adjacent data access requests are consistent. In response to determining that the hash values ​​of the data object replicas corresponding to the two adjacent data access requests are consistent, it is determined that the contents of the data object replicas corresponding to the two adjacent data access requests are identical, and the two adjacent data access requests perform the same access operation on the data object.

[0076] S302: In response to determining that two adjacent data access requests perform the same access operation on a data object, merge and de-duplicate data object copies corresponding to the two adjacent data access requests.

[0077] When two adjacent data access requests perform the same access operation on a data object, the contents of the corresponding generated data object copies are also duplicated, and deduplication processing can be performed, thereby reducing the data processing workload of the solution.

[0078] As shown in Figure 4, for data object Object2_v0 on storage device m2, the content of the data object replicas Object2_v1+Object2_v2 and Object2_v2 generated in response to data access requests from different clients may be duplicated. For example, if T1 and T2 modify the same content of Object2, the hash algorithm SHA-2 can be used to calculate a hash value for the two replicas. If the hash values ​​of the two replicas are determined to be the same, the contents of the two data object replicas are compared. If the contents are still the same, only one data object replica is created, and the access record data m1:o2:v1 and the access record data m1:o2:v2 both point to this replica.

[0079] S3: Determine result status information of at least one data access request, and merge multiple data object copies based on the result status information to determine access result data corresponding to the target data object.

[0080] After determining the data object replicas for a data object, the multiple data object replicas can be merged based on the access operation results of the corresponding multiple data access requests to determine the access result for the data object. The result status information is used to represent the access operation result of the data access request for the data object, including both successful and failed data access operations.

[0081] As shown in FIG6 , in a distributed storage system data access control method provided by one or more embodiments of the present application, merging multiple data object copies corresponding to a data object based on result status information includes:

[0082] S401: Determine in sequence whether a data object copy corresponding to at least one data access request is valid based on result status information of at least one data access request corresponding to a target data object.

[0083] The result status information includes whether the data access operation succeeded or failed. In some embodiments, based on the result status information of at least one data access request corresponding to the target data object, determining whether the data object copy corresponding to at least one data access request is valid can be performed using the following method:

[0084] In response to the result status information of the current data access request indicating that the data access operation is successful, the corresponding data object copy is marked as a valid data object copy.

[0085] In response to the result status information of the current data access request being that the data access operation fails, the corresponding data object copy and other associated data object copies are marked as invalid data object copies.

[0086] In the process of generating data copy objects, corresponding data object copies will be generated for the two different result states of the data access operation corresponding to the previous access record data. Therefore, the multiple data copy objects generated can cover all possible operation results of multiple data access requests.

[0087] When it is necessary to determine the access result for the target data object, the operation result that matches the actual access operation is selected from all possible results according to the actual data access operation result, thereby determining accurate result data.

[0088] S402: Select valid data object copies from multiple data object copies to merge, and determine the merge result as access result data to update the target data object.

[0089] Starting from the data object copy corresponding to the first data access request for the target data object, valid data object copies are selected from the corresponding data object copies for each data access request and merged one by one, and the final merged result is used as the access result for the data object.

[0090] As shown in Figure 7-a, in a distributed storage system, clients (T1 and T2) issue data access requests for five data objects (m1:o1, m1:o2, m2:o1, m2:o2, and m3:o1) located on three storage devices (machine1, machine2, and machine3, also referred to as m1, m2, and m3). The following describes the merging process for data object replicas, taking the case where the data access request from client T1 fails while the data access request from client T2 succeeds.

[0091] T1's data access request comes first. First, the result status information for T1's corresponding data access request is determined. In this example, the data access request from T1 fails, possibly because client T1 lacks permission or other prerequisites are not met. The data object replica corresponding to T1's data access request is set to an invalid data object replica (marked with an "×" symbol in Figure 7-a). T2's data access request comes later, and its corresponding result status information indicates that the data access request was successful, and the corresponding data object replica is a valid data object replica (not specifically marked in Figure 7-a).

[0092] For data object Object2_v0 on storage device m2, its corresponding data object replicas include Object2_v1, which corresponds to the data access request issued by T1, and Object2_v1+Object2_v2 and Object2_v2, which correspond to the data access request issued by T2. If the data access request from client T1 fails, data object replica Object2_v1 and its associated data object replicas Object2_v1+Object2_v2 can be marked as invalid data object replicas. If the data access request from client T2 succeeds, data object replica Object2_v2 can be marked as valid data object replica.

[0093] When merging multiple data object copies of the data object Object2_v0 of the storage device m2, a valid data object copy Object2_v2 is selected and merged with the data object Object2_v0, and the merged result is Object2_v2.

[0094] The data object copies corresponding to other multiple data objects are merged in the same way, and the final merge result is shown in Figure 7-b.

[0095] As shown in Figure 8-a, in a distributed storage system, clients (T1 and T2) issue data access requests for five data objects (m1:o1, m1:o2, m2:o1, m2:o2, and m3:o1) located on three storage devices (machine1, machine2, and machine3, also referred to as m1, m2, and m3). The following describes the merging process for data object replicas, taking the case where the data access request from client T1 succeeds while the data access request from client T2 fails.

[0096] T1's data access request comes first. The result status information for T1's data access request is first determined. In this example, if T1's data access request succeeds, the data object replica corresponding to T1's data access request is set to a valid data object replica (not specifically marked in Figure 8-a). T2's data access request comes later, and its corresponding result status information indicates that the data access request failed, and the corresponding data object replica is set to an invalid data object replica (marked with an "×" symbol in Figure 8-a).

[0097] For data object Object2_v0 on storage device m2, its corresponding data object replicas include Object2_v1, corresponding to the data access request issued by T1, and Object2_v1+Object2_v2 and Object2_v2, corresponding to the data access request issued by T2. If the data access request from client T1 succeeds, data object replica Object2_v1 can be marked as a valid data object replica. If the data access request from client T2 fails, data object replicas Object2_v1+Object2_v2 and Object2_v2 can be marked as invalid data object replicas.

[0098] When merging multiple data object copies of the data object Object2_v0 of the storage device m2, the valid data object copy Object2_v1 is selected and merged with the data object Object2_v0, and the merged result is Object2_v1.

[0099] The data object copies corresponding to other multiple data objects are merged in the same way, and the final merge result is shown in Figure 8-b.

[0100] In a distributed storage system, clients (T1 and T2) issue data access requests to five data objects (m1:o1, m1:o2, m2:o1, m2:o2, and m3:o1) located on three storage devices (machine1, machine2, and machine3, also referred to as m1, m2, and m3). Upon determining that the data access requests from clients T1 and T2 have failed, all data object replicas are deleted, and the data contents of the data objects remain unmarked. Upon determining that the data access requests from clients T1 and T2 have succeeded, the data object replicas corresponding to the successful results are selected from the data object replicas corresponding to each access record data and merged.

[0101] The above operation process for the four situations is a combined processing of the two operations T1 and T2. More data access requests also perform access control operations according to the same method and principle.

[0102] It should be noted that the data access control operation for each data object can be executed using the above-mentioned asynchronous processing method. First, a corresponding data object copy is generated based on at least one access record data corresponding to the data object, and then the data object copies are merged to determine the access operation result.

[0103] S4: Update the target data object according to the access result data.

[0104] The target data object is updated according to the access result data, and the data content of the target data object can be fed back to the corresponding client after the update.

[0105] For each data access request, valid data object copies are selected from the corresponding data object copies and merged one by one. The final merged result is used as the access result for the target data object, and the data target object is updated according to the determined access data result.

[0106] A distributed storage system data access control method receives one or more data access requests for a target data object and generates access record data corresponding to the data access requests. For at least one access request data item corresponding to the target data object, multiple data object copies are generated. These multiple data object copies are then merged based on actual operation result status information to determine accurate access result data for updating the target data object. This approach eliminates the need for a distributed locking mechanism and can accurately determine data operation results based on actual operation conditions. While ensuring data consistency in the distributed storage system, it can also effectively avoid the impact of the distributed locking mechanism on distributed storage data access performance optimization, further optimizing system performance.

[0107] Taking into account the situation that during the actual implementation of the actual plan, there may be too many data access requests for the same data object, or too many data object copies generated for the data object, which will affect the overall storage operation efficiency of the system, corresponding thresholds can be set to limit the number of data access requests, the corresponding number of access record data, and the number of data object copies of the same data object, and batch processing can be adopted to improve the overall storage operation efficiency of the system.

[0108] One or more embodiments of the present application provide a distributed storage system data access control method, which, after generating corresponding access record data for a data access request, further includes:

[0109] Determine whether the number of access record data items corresponding to the target data object exceeds a preset access threshold, where the preset access threshold can be flexibly set according to actual needs and the overall processing performance of the distributed system.

[0110] In response to the number of access record data items corresponding to the target data object exceeding a preset access threshold, a proxy node is set to process the multiple access record data items in batches.

[0111] As shown in FIG9 , in a distributed storage system data access control method provided by one or more embodiments of the present application, a proxy node is set to process multiple access record data in batches, including:

[0112] S501: Grouping multiple access record data items according to the time sequence of corresponding data access requests, wherein the number of access record data items in each group is lower than a preset access threshold.

[0113] The preset access threshold can be set to 10, for example. When the number of access record data items for a data object exceeds 10, the multiple access record data items can be sorted according to the chronological order of the corresponding data access requests, and all access record data items can be grouped sequentially so that the number of items in each group does not exceed 10. Various grouping methods can be adopted. For example, Anshun selects 9 access record data items per group, and the remaining multiple access record data items with less than 9 items are used as the final group. Another method is to divide the access record data equally, selecting an appropriate divisor based on the total number of items, and dividing the data equally according to this constraint, ensuring that the number of items in each group does not exceed 10 after the equal division.

[0114] S502: Generate multiple data object copies for the target data object according to the multiple access record data in each group.

[0115] S503: Merge multiple data object copies to determine a merge result, and update the data content of the target data object to the merge result.

[0116] First, multiple data access records in the first group are processed to generate multiple data object copies for the data object. The data object copies corresponding to the multiple data access records in the first group are then merged to obtain an intermediate merged result, and the content of the data object is updated to the intermediate merged result. Subsequently, based on the updated data object, the multiple data access records in the second group are processed to generate corresponding data object copies, which are then merged and updated until all access record data are processed.

[0117] As shown in FIG10 , in a distributed storage system data access control method provided by one or more embodiments of the present application, when generating a corresponding data object copy for a target data object based on access record data, the method further includes:

[0118] S601: Monitor the number of generated data object copies to determine whether the number of data object copies reaches a preset copy threshold.

[0119] The preset replica threshold can be flexibly set according to actual needs and the overall processing performance of the distributed system.

[0120] S602: In response to the number of data object copies reaching a preset copy threshold, stop generating new data object copies.

[0121] S603: Merge the result status information corresponding to the multiple generated data object copies to determine a merge result, and update the data content of the data object to the merge result.

[0122] S604: Clear the number of data object copies and restart statistical monitoring until corresponding data object copies are generated for all access record data.

[0123] The preset copy threshold can be set to 16, for example. For a data object, when the number of generated data object copies reaches 16, the generation of new data object copies is suspended. A merge operation is first performed on the 16 data object copies that have already been generated, and the merge result is updated to the data object, and the 16 generated data object copies are deleted. Thereafter, the statistical data is cleared and the generation of new data object copies continues. In response to determining that the number of data object copies has again reached the threshold, the generation of new data object copies is again suspended, and the merge and update operation is performed again on the 16 newly generated data object copies. This continues until all access record data corresponds to multiple data object copies generated. The last multiple data object copies generated are merged, and the merge result is updated to the data object.

[0124] This approach can ensure that when processing data access operations for data access requests, the number of data object copies generated does not exceed the preset threshold, thereby reducing the amount of data access operation processing and improving the execution efficiency of the overall data processing flow.

[0125] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0126] It should be noted that the above description is of specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0127] Based on the same purpose, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides a distributed storage system data access control device.

[0128] Referring to FIG11 , a distributed storage system data access control device includes:

[0129] An access record module is used to obtain at least one data access request for a target data object and generate corresponding access record data for the data access request; a copy generation module is used to generate a corresponding data object copy for the target data object based on the access record data; a copy merging module is used to determine the result status information of at least one data access request and merge multiple data object copies based on the result status information to determine the access result data corresponding to the target data object; and a data update module is used to update the target data object based on the access result data.

[0130] In a distributed storage system data access control device provided by one or more embodiments of the present application, the access record module is also used to extract access operation information for the target data object from the data access request; and generate access record data corresponding to the target data object based on the access operation information.

[0131] In a distributed storage system data access control device provided by one or more embodiments of the present application, after the access record module generates corresponding access record data for a data access request, it is also used to store the access record data in a pre-write log, and the pre-write log is set in a millisecond-level storage medium.

[0132] In a distributed storage system data access control device provided by one or more embodiments of the present application, the copy generation module is also used to generate multiple data object copies for the target data object in sequence according to the order of request time of at least one data access request corresponding to at least one access record data.

[0133] In a distributed storage system data access control device provided by one or more embodiments of the present application, the copy generation module is also used to determine a data object copy corresponding to the previous access record data of the current access record data; for two different result states of the data access operation corresponding to the previous access record data, corresponding data object copies are generated according to the access operation information in the current access record data; wherein the result state includes a successful data access operation and a failed data access operation.

[0134] In a distributed storage system data access control device provided by one or more embodiments of the present application, the copy generation module is also used to respond to the data access request corresponding to the current access record data as the first data access request for the data object, and for the initial data content in the data object, generate a data object copy corresponding to the initial data content according to the access operation information in the current access record data.

[0135] In a distributed storage system data access control device provided by one or more embodiments of the present application, a target data object and the corresponding multiple data object copies are connected by a bidirectional linked list, and a target data object and the multiple data object copies corresponding to the same access record data are connected by a linked list; a local lock is set in the memory corresponding to the target data object, and the local lock is used to maintain the bidirectional linked list between the target data object and the corresponding multiple data object copies.

[0136] In a distributed storage system data access control device provided by one or more embodiments of the present application, the copy merging module is also used to determine whether the data object copy corresponding to at least one data access request is valid in turn based on the result status information of at least one data access request corresponding to the target data object; select a valid data object copy from multiple data object copies for merging, and determine the merging result as the access result data to update the target data object.

[0137] In a distributed storage system data access control device provided by one or more embodiments of the present application, result status information includes data access operation success and data access operation failure. The replica merging module is further configured to, in response to determining that the result status information of the current data access request is data access operation success, mark the data object replica corresponding to the target data object as a valid data object replica; and, in response to determining that the result status information of the current data access request is data access operation failure, mark the data object replica corresponding to the target data object and other associated data object replicas as invalid data object replicas.

[0138] In a distributed storage system data access control device provided by one or more embodiments of the present application, the replica merging module is further used to merge valid data object replicas corresponding to at least one data access request corresponding to the target data object in sequence.

[0139] In a distributed storage system data access control device provided by one or more embodiments of the present application, the copy merging module is further used to compare the contents of the data object copies corresponding to two adjacent data access requests before merging multiple data object copies based on result status information to determine whether the two adjacent data access requests perform the same access operation on the data object; in response to determining that the two adjacent data access requests perform the same access operation on the data object, the data object copies corresponding to the two adjacent data access requests are merged and deduplicated.

[0140] In a distributed storage system data access control device provided by one or more embodiments of the present application, the replica merging module is also used to calculate hash values ​​for data object replicas corresponding to two adjacent data access requests to determine whether the hash values ​​of the data object replicas corresponding to the two adjacent data access requests are consistent; in response to determining that the hash values ​​of the data object replicas in response to the two adjacent data access requests are consistent, it is determined that the contents of the data object replicas corresponding to the two adjacent data access requests are the same, and the two adjacent data access requests perform the same access operation on the data object.

[0141] In a distributed storage system data access control device provided by one or more embodiments of the present application, after the access record module generates corresponding access record data for a data access request, it is also used to determine whether the number of access record data items corresponding to the target data object exceeds the preset access threshold; in response to determining that the number of access record data items corresponding to the target data object exceeds the preset access threshold, a proxy node is set to process multiple access record data in batches.

[0142] In a distributed storage system data access control device provided by one or more embodiments of the present application, the access record module is further used to group multiple access record data in chronological order of the corresponding data access requests, where the number of access record data items in each group is lower than a preset access threshold; generate multiple data object copies for the target data object based on the multiple access record data in each group; merge the multiple data object copies to determine the merge result, and update the data content of the target data object to the merge result.

[0143] In a distributed storage system data access control device provided by one or more embodiments of the present application, when a copy generation module generates corresponding data object copies for a target data object based on access record data, the copy generation module is also used to monitor the number of generated data object copies to determine whether the number of data object copies reaches a preset copy threshold; in response to determining that the number of data object copies reaches the preset copy threshold, suspend the generation of new data object copies; merge the corresponding result status information of the multiple generated data object copies to determine the merge result, and update the data content of the data object to the merge result; reset the number of data object copies and restart statistical monitoring until corresponding data object copies are generated for all access record data.

[0144] One or more embodiments of the present application provide a distributed storage system data access control device that executes multiple data access operations on multiple target data objects in a synchronous parallel processing manner; and executes data access operations on the same target data object in an asynchronous processing manner.

[0145] In a distributed storage system data access control device provided by one or more embodiments of the present application, a target data object and a corresponding data object copy are set in a storage memory of the distributed system.

[0146] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing one or more embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0147] The apparatus of the above embodiment is used to implement the corresponding method in the above embodiment and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0148] FIG12 is a schematic diagram showing the hardware structure of an electronic device provided in this embodiment. The device may include: one or more processors 1010, a memory 1020 associated with the one or more processors 1010, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processors 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0149] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0150] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present application are implemented through software or firmware, the relevant computer-readable instructions are stored in the memory 1020 and read and executed by the processor 1010.

[0151] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0152] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means, such as USB (Universal Serial Bus), network cables, etc., or wireless means, such as mobile networks, Wi-Fi (wireless communication network technology), Bluetooth, etc.

[0153] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0154] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, during implementation, the device may also include other components required for normal operation. In addition, those skilled in the art will understand that the above device may also include only the components required to implement the embodiments of the present application, and does not necessarily include all the components shown in the figure.

[0155] The electronic devices of the above embodiments are used to implement the corresponding methods in the above embodiments and have the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0156] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, as shown in Figure 13, the non-transitory computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the distributed storage system data access control method of any of the above embodiments is implemented.

[0157] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM, SRAM), static random access memory (Static RAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape-type disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0158] The computer-readable instructions stored in the storage medium of the above embodiment are used to enable a computer to execute the distributed storage system data access control method of any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0159] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a mechanical hard disk, or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0160] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. In some embodiments, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0161] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-readable instructions.

[0163] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0164] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0165] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment.

[0166] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0167] Although the present disclosure has been described in conjunction with the embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0168] The one or more embodiments of the present application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the one or more embodiments of the present application should be included within the scope of protection of this disclosure.

Claims

1. A distributed storage system data access control method, characterized in that: include: Obtain at least one data access request for a target data object, and generate corresponding access record data for each data access request; generating a corresponding plurality of data object copies for the target data object according to the access record data; Determine result status information of the at least one data access request, and merge the multiple data object copies based on the result status information to determine access result data corresponding to the target data object; as well as The target data object is updated according to the access result data.

2. The method according to claim 1, characterized in that The step of generating corresponding access record data for each data access request includes: Extracting access operation information for the target data object from each data access request; and The access record data corresponding to the target data object is generated according to the access operation information.

3. The method according to claim 1, characterized in that After the step of generating corresponding access record data for each data access request, the method further includes storing the access record data in a write-ahead log, wherein the write-ahead log is set in a millisecond-level storage medium.

4. The method according to claim 1, characterized in that: The step of generating a corresponding plurality of data object copies for the target data object according to the access record data comprises: The multiple data object copies are generated for the target data object in sequence according to the order of request time of at least one data access request corresponding to at least one access record data.

5. The method according to claim 4, characterized in that The step of sequentially generating the plurality of data object copies for the target data object comprises: Determine a copy of the data object corresponding to the previous access record data of the current access record data; and For two different result states of the data access operation corresponding to the previous access record data, generate corresponding data object copies according to the access operation information in the current access record data; The result status includes data access operation success and data access operation failure.

6. The method according to claim 5, characterized in that The step of generating a plurality of corresponding data object copies for the data object according to the access record data comprises: In response to the data access request corresponding to the current access record data being the first data access request for the data object, for the initial data content in the data object, a data object copy corresponding to the initial data content is generated according to the access operation information in the current access record data.

7. The method according to claim 4, characterized in that The method further comprises: The target data object and the corresponding multiple data object copies are connected by a bidirectional linked list, and the target data object and the multiple data object copies corresponding to the same access record data are connected by a linked list; and The method further comprises: A local lock is set in the memory corresponding to the target data object to maintain the bidirectional linked list between the target data object and the corresponding multiple data object copies.

8. The method according to claim 4, characterized in that The step of merging the multiple data object copies based on the result status information includes: determining in sequence whether the data object copy corresponding to the at least one data access request is valid according to the result status information of the at least one data access request corresponding to the target data object; and A valid data object copy is selected from the multiple data object copies for merging, and a merging result is determined as the access result data to be updated to the target data object.

9. The method according to claim 8, characterized in that The result status information includes data access operation success and data access operation failure; The step of determining in sequence whether the data object copy corresponding to the at least one data access request is valid according to the result status information of the at least one data access request corresponding to the target data object comprises: In response to the result status information of the current data access request indicating that the data access operation is successful, marking the data object copy corresponding to the current data access request as a valid data object copy; or In response to the result status information of the current data access request indicating that the data access operation fails, the data object copy corresponding to the current data access request and other associated data object copies are marked as invalid data object copies.

10. The method according to claim 9, characterized in that The step of selecting a valid data object copy from the multiple data object copies for merging includes: For the at least one data access request corresponding to the target data object, the valid data object copies corresponding to the at least one data access request are merged in sequence.

11. The method according to claim 1, characterized in that: Before the step of merging the multiple data object copies based on the result status information, the method further includes: Comparing the contents of the data object copies corresponding to two adjacent data access requests to determine whether the two adjacent data access requests perform the same access operation on the data object; and In response to determining that the two adjacent data access requests perform the same access operation on the data object, the data object copies corresponding to the two adjacent data access requests are merged and deduplicated.

12. The method according to claim 11, characterized in that The step of comparing the contents of the data object copies corresponding to the two adjacent data access requests includes: Calculating hash values ​​for the data object copies corresponding to the two adjacent data access requests to determine whether the hash values ​​of the data object copies corresponding to the two adjacent data access requests are consistent; and In response to determining that the hash values ​​of the data object copies corresponding to the two adjacent data access requests are consistent, it is determined that the data object copies corresponding to the two adjacent data access requests have the same content, and the two adjacent data access requests perform the same access operation on the data object.

13. The method according to claim 8, characterized in that After the step of generating corresponding access record data for each data access request, the method further includes: Determining whether the number of access record data items corresponding to the target data object exceeds a preset access threshold; and In response to the number of the access record data items corresponding to the target data object exceeding the preset access threshold, setting the proxy node to process the multiple access record data in batches.

14. The method according to claim 13, characterized in that The step of setting the proxy node to process the multiple access record data in batches includes: The plurality of access record data are grouped according to the time sequence of the corresponding data access requests, and the number of access record data items in each group is lower than the preset access threshold; Generate the multiple data object copies for the target data object according to the multiple access record data in each group in sequence; and The multiple data object copies are merged to determine a merge result, and the data content of the target data object is updated to the merge result.

15. The method according to claim 1, characterized in that The step of generating a plurality of corresponding data object copies for the target data object according to the access record data further includes: Monitoring the number of the plurality of data object copies to determine whether the number of the plurality of data object copies reaches a preset copy threshold; In response to the number of the plurality of data object copies reaching the preset copy threshold, suspending the generation of new data object copies; Merging corresponding result status information of the generated multiple data object copies to determine a merge result, and updating the data content of the data object to the merge result; and The number of data object copies is cleared and statistical monitoring is restarted until corresponding data object copies are generated for all the access record data.

16. The method according to claim 1, characterized in that Multiple data access operations for multiple target data objects are performed in a synchronous parallel processing manner; and Data access operations on the same target data object are executed in an asynchronous processing manner.

17. The method according to claim 1, characterized in that The method is applied to a distributed storage system, and the target data object and corresponding multiple data object copies are set in the storage memory of the distributed storage system.

18. A distributed storage system data access control device, characterized in that: include: An access record module, used to obtain at least one data access request for a target data object, and generate corresponding access record data for each data access request; A copy generation module, used to generate corresponding multiple data object copies for the target data object according to the access record data; A replica merging module, configured to determine result status information of at least one of the data access requests, and merge the multiple data object replicas based on the result status information to determine access result data corresponding to the target data object; as well as A data updating module is used to update the target data object according to the access result data.

19. An electronic device, characterized in that: include one or more processors; and A memory associated with the one or more processors, the memory being used to store computer-readable instructions, wherein the computer-readable instructions implement the method according to any one of claims 1 to 17 when read and executed by the one or more processors.

20. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the method according to any one of claims 1 to 17 is implemented.

Images