A data access system, method, apparatus and device

Abstract

A data access system, method, device and equipment, in the application, the first client receives a data write request, the data write request is used to request to write target data; the first client caches a copy of the target data, and calls an erasure code to obtain M shards of the target data, M is a positive integer, and the first client instructs a storage cluster to store the M shards. The storage cluster stores the M shards under the instruction of the first client; when the first client determines that m shards in the M shards have been stored to the storage cluster, the first client feeds back a data write response, the data write response indicates that the target data is written successfully, and m is a positive integer less than M. The first client can feed back the data write response as long as it is determined that m shards have been stored to the storage cluster, thereby effectively reducing the data write delay. The first client side caches a copy of the target data, ensures that at least one complete copy of the target data is stored in the system, and ensures the reliability of the target data.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data access system, method, apparatus and device. Background Technology

[0002] A distributed system is a computer system composed of multiple independent nodes that communicate and coordinate through a network to achieve a common goal. In a distributed system, different nodes can perform independent computation and processing tasks simultaneously, communicating with each other through message passing.

[0003] Distributed systems need to ensure data storage fault tolerance, meaning that data is not lost even if some nodes fail. Data fault tolerance is achieved through data redundancy, with multiple replicas being a commonly used method.

[0004] In a multi-replica mechanism, n replicas of the data to be written to the distributed system can be created. Each replica is identical to the original number of replicas, and each of the n nodes in the distributed system stores one replica. The redundancy of the multi-replica mechanism is n-1, meaning it can tolerate the failure of n-1 nodes. Even if n-1 nodes fail, data can still be read from the distributed system.

[0005] However, in the multi-replica mechanism, a replica needs to be transmitted to each of the n nodes. This results in a large consumption of network bandwidth in the distributed system. Furthermore, storing n replicas of data in the n nodes requires a significant amount of storage space in the distributed system. Therefore, the multi-replica mechanism consumes a large amount of network bandwidth and storage space. Summary of the Invention

[0006] This application provides a data access system, method, apparatus, and device to reduce the bandwidth and storage space consumption of data writing.

[0007] In a first aspect, embodiments of this application provide a data access system, which includes at least one client and a storage cluster. The following description uses a first client among the at least one clients as an example. This first client corresponds to the client in the disclosure document that handles data write requests.

[0008] The first client receives a data write request, which requests the writing of target data. The first client caches a copy of the target data and invokes erasure coding to obtain M shards of the target data. The first client instructs the storage cluster to store these M shards. M is a positive integer. The M shards include m data shards and k parity shards, where m and k are less than M, and both m and k are positive integers. That is, M is numerically equal to m + k.

[0009] The storage cluster stores M shards under the instruction of the first client; when the first client determines that m shards out of the M shards have been stored in the storage cluster, it sends a data write response, which indicates that the target data has been successfully written.

[0010] Through the aforementioned system, the first client uses erasure coding to generate multiple fragments of the target data during the data write request process. It only needs to interact with the storage cluster regarding these fragments, effectively reducing bandwidth and storage space consumption during data writes. Furthermore, the first client can respond to the data write request as soon as it confirms that m fragments have been stored in the storage cluster, effectively reducing data write latency. In addition, the first client caches a copy of the target data, ensuring that at least one complete copy of the target data is stored in the system, thus guaranteeing the reliability of the target data.

[0011] In one possible implementation, the first client, after determining that all M shards are stored in the storage cluster, deletes the cached copy of the target data and sends back a data write response.

[0012] With the above system, all M shards are stored in the storage cluster, indicating that the shards stored in the storage cluster are sufficient to ensure the reliability of the target data. The first client deletes the copy of the target data to save storage space on the first client side.

[0013] In one possible implementation, the client can also handle data read requests. To highlight the advantage of the client handling data read requests in this application, the second client among the at least one clients is taken as an example. The second client can be understood as any other client besides the first client. The second client corresponds to the client that handles data read requests in the embodiments of this application.

[0014] The second client receives a data read request, which is used to request the reading of target data; the second client obtains a copy of the target data from the first client; and sends back a data read response, which carries a copy of the target data.

[0015] Through the above system, clients can interact to obtain copies of the target data, ensuring that the second client can obtain the target data in a timely manner and then provide a data reading response, thereby improving data reading efficiency.

[0016] In one possible implementation, if the second client fails to obtain m shards of the target data from the storage cluster, the second client obtains a copy of the target data from the first client.

[0017] Through the above system, if the second client fails to obtain m shards of the target data from the storage cluster, it indicates that the shards of the target data may not be stored in the storage cluster. The second client can interact with the first client to obtain a copy of the target data, ensuring that the second client can eventually obtain the target data and increasing the probability of successfully reading the target data.

[0018] In one possible implementation, a data write request is used to request the writing of first data, which includes multiple target data. When m shards of each target data have been stored in the storage cluster, the first client sends back a data write response, which indicates that the first data has been successfully written.

[0019] With the above system, when storing the first data, the first data can be divided into multiple target data, and then each target data fragment can be stored separately to ensure that the first data can be stored completely in the storage cluster.

[0020] In one possible implementation, the storage cluster includes multiple fault-tolerant groups, each of which includes M storage nodes. The first client determines a target fault-tolerant group from among the multiple fault-tolerant groups and sends data write instructions to the M storage nodes in the target fault-tolerant group. The data write instructions are used to request the writing of a shard of the target data. The shards requested to be written are different for the data write instructions sent to different storage nodes.

[0021] The above system allows the first client to easily determine the storage node of the shard to which the target data needs to be written by selecting a fault-tolerant group.

[0022] In one possible implementation, the first client receives write success responses from m storage nodes in the target fault-tolerant group, which indicate that the shard write was successful.

[0023] Through the above system, the first client receives write success responses from m storage nodes, indicating that the m storage nodes have stored the fragments of the target data, and the first client can send a data write response.

[0024] In one possible implementation, the second client interacts with the target fault-tolerant group in one or more rounds, but fails to acquire m of the M shards.

[0025] Through the above system, the second client can increase the probability of obtaining the target data by interacting with the target fault-tolerant group multiple times. If m fragments are not obtained after multiple interactions, it indicates that the target data has not been stored in the target fault-tolerant group.

[0026] In one possible implementation, the second client obtains information about the first client from the storage cluster (such as the target fault-tolerant group), where the information about the first client is its identifier or address; the second client then obtains a copy of the target data from the first client based on the first client's information.

[0027] Through the above system, the second client obtains information from the first client to ensure that it obtains a copy of the target data from the first client.

[0028] In one possible implementation, any storage node in the target fault-tolerant group receives a data write instruction, stores a fragment of the target data according to the data write instruction, and records the log information of the target data, including the version number of the fragment and the information of the first client that requested to write the fragment.

[0029] Through the above system, in addition to storing fragments of the target data, the storage node can also record relevant information about the target data, which facilitates subsequent version rollback of the target data or provides information of the first client to the second client.

[0030] In one possible implementation, the second client interacts with the target fault tolerance group at least once. "Failed to acquire m shards out of M shards" means that the second client, after at least one round of interaction with the target fault tolerance group, failed to acquire m shards with the same version number as the target data.

[0031] With the above system, since the target data may be updated multiple times, the second client needs to ensure that the version numbers of the target data fragments are consistent when obtaining the target data fragments. If the version numbers of the obtained target data fragments are inconsistent, it can be considered that the target data has not been obtained.

[0032] Secondly, this application also provides a data access method, the relevant details of which can be found in the foregoing description and will not be repeated here. In this method:

[0033] The first client receives a data write request, which is used to request the writing of target data.

[0034] After receiving the data write request, the first client caches a copy of the target data and calls erasure coding to obtain M fragments of the target data. These M fragments include m data fragments and k parity fragments, where M is a positive integer, m and k are both less than M, and m and k are both positive integers.

[0035] The first client instructs the storage cluster to store M shards. Once it determines that m of the M shards have been stored in the storage cluster, it sends a data write response, indicating that the target data has been successfully written.

[0036] In one possible implementation, the first client, after determining that all M shards are stored in the storage cluster, deletes the cached copy of the target data and sends back a data write response.

[0037] In one possible implementation, the second client receives a data read request, which requests the reading of target data. The second client obtains a copy of the target data from the first client. The second client then sends back a data read response, which carries a copy of the target data.

[0038] In one possible implementation, the second client obtains a copy of the target data from the first client from m shards of the storage cluster that do not contain the target data.

[0039] In one possible implementation, a data write request is used to request the writing of first data, which includes multiple target data. When m shards of each target data have been stored in the storage cluster, the first client sends back a data write response, which indicates that the first data has been successfully written.

[0040] In one possible implementation, the storage cluster includes multiple fault-tolerant groups, each of which includes M storage nodes. When the first client instructs the storage cluster to store M shards, the first client determines the target fault-tolerant group from the multiple fault-tolerant groups.

[0041] The first client sends data write instructions to each of the M storage nodes in the target fault-tolerant group. The data write instructions are used to request the writing of a fragment of the target data. The data write instructions sent to different storage nodes request different fragments to be written.

[0042] In one possible implementation, the first client receives write success responses from m storage nodes in the target fault-tolerant group. These write success responses indicate that the shard write was successful. The first client then determines that m shards out of the M shards have been stored in the storage cluster.

[0043] In one possible implementation, the second client interacts with the target fault-tolerant group at least once, but fails to acquire m of the M shards.

[0044] In one possible implementation, when the second client obtains a copy of the target data from the first client, the second client also obtains information about the first client from the target fault-tolerant group. This information includes the first client's identifier or address. The second client then obtains the copy of the target data from the first client based on this information.

[0045] In one possible implementation, any storage node in the target fault-tolerant group receives a data write instruction, stores a fragment of the target data according to the data write instruction, and records the log information of the target data, including the version number of the fragment and the information of the first client that requested to write the fragment.

[0046] In one possible implementation, the second client interacts with the target fault-tolerant group at least once. "Not acquiring m shards out of M shards" means that the second client, after at least one round of interaction with the target fault-tolerant group, has not acquired m shards with the same version number as the target data.

[0047] Thirdly, this application also provides a data writing method, which can be executed by a client. Related useful information can be found in the foregoing description and will not be repeated here. In this method:

[0048] The client receives a data write request, which requests the writing of target data. A copy of the target data is cached, and erasure coding is invoked to obtain M fragments of the target data. These M fragments include m data fragments and k parity fragments, where M is a positive integer, and both m and k are less than M and are positive integers.

[0049] The client instructs the storage cluster to store M shards. Once it determines that m of the M shards have been stored in the storage cluster, it sends a data write response, indicating that the target data has been successfully written.

[0050] In one possible implementation, if the client determines that all M shards are stored in the storage cluster, it deletes the cached copy of the target data and sends a data write response again.

[0051] In one possible implementation, the client receives a data read request, which requests the reading of target data. The client obtains a copy of the target data from another client; the client then sends back a data read response, which carries a copy of the target data.

[0052] In one possible implementation, if the client does not obtain m shards of the target data from the storage cluster, it obtains a copy of the target data from another client.

[0053] In one possible implementation, a data write request is used to request the writing of first data, which includes multiple target data. When m shards of each target data have been stored in the storage cluster, the client sends a data write response, which indicates that the first data has been successfully written.

[0054] In one possible implementation, the storage cluster includes multiple fault-tolerant groups, each containing M storage nodes. The client determines the target fault-tolerant group from among the multiple fault-tolerant groups. The client sends data write instructions to the M storage nodes in the target fault-tolerant group, respectively. The data write instructions are used to request the writing of a shard of the target data. The shards requested to be written are different for the data write instructions sent to different storage nodes.

[0055] In one possible implementation, the client receives write success responses from m storage nodes in the target fault-tolerant group. These write success responses indicate that the shard write was successful. The client then determines that m shards out of the M shards have been stored in the storage cluster.

[0056] In one possible implementation, the client interacts with the target fault-tolerant group at least once, but fails to acquire m of the M shards.

[0057] In one possible implementation, the client obtains information about the first client from the target fault-tolerant group, where the information is the first client's identifier or address information; the client obtains copies of the target data from other clients based on the information of other clients.

[0058] In one possible implementation, the client failing to acquire m of the M shards means that the client has interacted with the target fault-tolerant group at least once and has not acquired m shards with the same version number as the target data.

[0059] Fourthly, embodiments of this application also provide a data access device. This main device has the function of implementing the behavior of the first client or the second client in the method examples of the second or third aspect described above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the data access device includes a receiving module, a processing module, and a sending module. These modules can perform the corresponding functions in the second aspect or the method examples of the second aspect described above. For details, please refer to the detailed description in the method examples, which will not be repeated here.

[0060] Fifthly, this application also provides a computing device, which includes a processor and a memory, and may further include a communication interface. The processor executes computer program instructions stored in the memory to perform the method provided by the second aspect or any possible implementation thereof, or to perform the method provided by the third aspect or any possible implementation thereof. The memory is coupled to the processor and stores computer program instructions and data necessary for data access. The communication interface is used for communicating with other devices.

[0061] Sixthly, this application provides a computing device system including at least one computing device. Each computing device includes a memory and a processor. The processor of at least one computing device is configured to access computer program instructions in the memory to execute the method provided by the second aspect or any possible implementation thereof, or to execute the method provided by the third aspect or any possible implementation thereof.

[0062] In a seventh aspect, this application provides a computer-readable storage medium, which, when executed by a computing device, allows the computing device to perform the method provided in the second aspect or any possible implementation thereof, or to perform the method provided in the third aspect or any possible implementation thereof.

[0063] The storage medium stores computer program instructions. This storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid state drive (SSD).

[0064] Eighthly, this application provides a computing device program product, which includes computer program instructions that, when executed by a computing device, enable the computing device to perform the method provided in the second aspect or any possible implementation thereof, or to perform the method provided in the third aspect or any possible implementation thereof.

[0065] The computer program product may be a software installation package, and when it is necessary to use the methods provided in the second aspect or any possible implementation of the second aspect, or the third aspect or any possible implementation of the third aspect, the computer program product may be downloaded and executed on a computing device.

[0066] Ninthly, this application also provides a computer chip connected to a memory, the chip being used to read and execute computer program instructions stored in the memory, to execute the methods in the second aspect and various possible implementations of the second aspect, or to execute the methods in the third aspect and various possible implementations of the third aspect.

[0067] For the technical effects that can be achieved by the second to ninth aspects mentioned above, please refer to the description of the technical effects that can be achieved by the corresponding design schemes in the first aspect mentioned above. This application will not repeat them here. Attached Figure Description

[0068] Figure 1 This application provides a schematic diagram of a data writing process using erasure coding, as illustrated in an embodiment of the present application.

[0069] Figure 2 This is a schematic diagram of the structure of a data access system provided in an embodiment of this application;

[0070] Figures 3A-3C This is a schematic diagram illustrating the deployment of a data access system provided in an embodiment of this application;

[0071] Figure 4 A schematic diagram of a data access method provided in this application;

[0072] Figure 5 A schematic diagram of the structure of a data access device provided in this application;

[0073] Figures 6-7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0074] Before introducing a data access system, method, apparatus, and device mentioned in the embodiments of this application, we will first explain an important concept involved in the embodiments of this application—erasure coding (EC) mechanism.

[0075] Erasure coding (EC) is a mechanism to ensure data reliability. EC involves dividing data into multiple data fragments, calculating a checksum fragment for each fragment using a specific algorithm, and then recovering the data from the lost fragments using the remaining fragments and the checksum fragment, thus ensuring data reliability.

[0076] In erasure coding, the original data is divided into m data fragments. An erasure coding algorithm is then called to generate k parity fragments based on the original data. These m data fragments and k parity fragments (a total of M fragments) are then stored. There are many types of erasure coding algorithms, such as Reed-Solomon (RS) coding. Here, M is a positive integer, and both m and k are less than M and are positive integers.

[0077] Erasure coding mechanisms can tolerate the loss or corruption of k fragments (which can be data fragments or parity fragments). That is, if k fragments out of M fragments are damaged or lost, the original data can be recovered using the remaining m fragments. If the number of damaged or lost fragments out of the M fragments exceeds k, the original data cannot be recovered. k can be considered the redundancy of the erasure coding mechanism.

[0078] It should be noted that M and k shards refer to the number of shards. The remaining m shards can be all data shards, or they can include both data shards and parity shards.

[0079] like Figure 1The diagram shows a write process using erasure coding. The system using erasure coding is a distributed system, which includes multiple storage nodes and clients.

[0080] The client can receive data write requests triggered by the user. The data write request is used to request the writing of target data. The client divides the target data into m data fragments and generates k verification fragments, that is, a total of M fragments.

[0081] The client sends data write instructions to M storage nodes respectively. The data write instructions are used to request the writing of a shard, which can be a data shard or a parity shard.

[0082] Upon receiving the data write instruction, any one of the M storage nodes stores the fragment requested by the data write instruction and sends a write success response to the client to notify the client that the fragment write was successful.

[0083] After receiving the data write response from the M storage nodes, the client determines that the target data has been successfully written and notifies the user that the target data has been successfully written.

[0084] In this write process, the client needs to wait for all M storage nodes to respond with a successful write response before it can confirm that the target data has been written successfully and then notify the user. If any of the M storage nodes fails to respond with a data write response in a timely manner, it will increase the latency of writing the target data and affect the writing efficiency.

[0085] In this embodiment, the client uses erasure coding to store the target data. After m out of M storage nodes have responded with a successful write response, the sender of the data write request is notified that the target data has been successfully written, thus ensuring the write latency of the target data. Additionally, the client can cache a copy of the target data to ensure its reliability.

[0086] like Figure 2 The diagram shown illustrates a data access system architecture applicable to an embodiment of this application. This data access system employs a distributed architecture, meaning that data can be distributed and stored across multiple storage nodes 210. The data access system includes a client 100 and a storage cluster 200. The storage cluster 200 includes multiple storage nodes 210.

[0087] Client 100 is the core of data processing in this data access system. Client 100 can receive and process data access requests. While processing the request, it interacts with storage nodes 210 in the storage cluster 200 to complete the data access. After completing the data access, it sends a data access response indicating the result. Multiple clients 100 can be deployed in this data access system, or only one client 100 can be deployed.

[0088] This application's embodiments involve two types of data access requests: one is a data access request for writing target data, which can also be called a data write request; the other is a data access request for reading target data, which can also be called a data read request. When a data access request requests to write target data, the data access response indicates that the target data has been successfully written; when a data access request requests to read target data, the data access response carries the target data.

[0089] It is worth noting that in the embodiments of this application, writing target data can be writing target data or updating target data that has already been written. That is, updating target data is also a data writing process, in which updated target data needs to be written.

[0090] To balance data write latency and data reliability, the client 100 employs erasure coding to complete the write of the target data when processing data write requests. Specifically, the client 100 distributes m data fragments and k parity fragments generated based on the target data across M storage nodes 210 of the storage cluster 200, with each storage node 210 storing one data fragment or one parity fragment. During this data write process, the client 100 possesses some or all of the following characteristics:

[0091] Feature 1: Feedback is provided when m shards are successfully written.

[0092] If the client 100 determines that m out of the M storage nodes 210 have successfully written data fragments or verification fragments, the client 100 can send back a data write response to indicate that the target data has been successfully written.

[0093] In other words, client 100 does not need to wait for all M storage nodes 210 to have successfully written data shards or checksum shards before sending a data write response. This allows for timely feedback of the data write response, enabling the initiator of the data write request to promptly know the write result of the target data without waiting for a long time, effectively reducing the write latency of the target data and improving the write efficiency of the target data.

[0094] Feature 2: Feedback the data write response again when all fragments are successfully written.

[0095] After the client 100 successfully writes to m shards, it can send a data write response. After that, the client 100 can continue to determine the write status of the data shards or parity shards in the remaining k storage nodes 210, that is, determine whether the data shards or parity shards in the remaining k storage nodes 210 have been successfully written.

[0096] If the data sharding or verification sharding is successfully written in the remaining k storage nodes 210, the client 100 can send a data write response again.

[0097] The successful writing of all data shards and verification shards indicates that the target data has been completely stored in storage cluster 200. Client 100 can send a data write response back to the initiator of the data write request so that the initiator can understand the true write status of the target data.

[0098] Feature 3: Data Replica Caching

[0099] Client 100 is configured with a cache space. After receiving a data write request, client 100 can store a copy of the target data in this cache space. It then distributes m data shards and k parity shards generated based on the target data across the M storage nodes 210 of storage cluster 200. Only after all data shards and parity shards have been successfully written does client 100 delete the copy of the target data stored in the cache space. In other words, client 100 will always retain a copy of the target data while all data shards and parity shards are not yet written to storage cluster 200.

[0100] When m data shards and k verification shards generated based on the target data are distributed and stored on M storage nodes 210 of the storage cluster 200, if other clients are reading the target data, the client 100 can provide a copy of the target data to other clients to ensure that other clients can read the target data.

[0101] To balance data read latency and data reliability, when processing a data read request, client 100 interacts with nodes in storage cluster 200 and obtains M data shards to verify the m shards. The target data is then generated using these m shards. During this data read process, client 100 possesses some or all of the following characteristics:

[0102] Feature 1: It interacts with the storage cluster 200 in multiple rounds to read m shards.

[0103] When client 100 retrieves target data from storage cluster 200, it can perform multiple rounds of interaction with storage cluster 200 to read m shards. The goal of each round of interaction is to obtain m shards from the M shards. In each round, client 100 can interact with some or all of the M storage nodes 210 to request the m shards.

[0104] If the client 100 fails to acquire the required m shards during the first round of interaction, the next round of interaction is executed until the client 100 acquires m shards. For example, during the first round of interaction, client 100 interacts with m of the M storage nodes 210 to request m shards. If client 100 fails to acquire m shards in the first round (e.g., due to a faulty storage node 210 or a storage node 210 that fails to provide shard information in a timely manner), client 100 begins the second round of interaction, interacting with the remaining k storage nodes, until the sum of the shards acquired in the second round and the shards acquired in the first round equals m.

[0105] Feature 2: Interact with client 100, which holds a copy of the target data, to obtain a copy of the target data.

[0106] Multiple clients 100 can be deployed in this data access system, which means that different clients 100 can access the same target data. For example, one client 100 can interact with the storage cluster 200 to write the target data according to the received data write request, and another client 100 can interact with the storage cluster 200 to read the target data according to the received data read request.

[0107] Given that client 100 possesses characteristic three during the writing of target data, a copy of the target data is always retained during the process of writing all M data shards to storage cluster 200.

[0108] If client 100 fails to obtain m shards from storage cluster 200 during the processing of a data read request, there may be three reasons:

[0109] Reason 1: Other clients are currently writing target data, and at least k+1 of the M data shards have not yet been written to storage cluster 200. In this case, other clients will retain copies of the target data, and client 100 can interact with these other clients to obtain copies of the target data.

[0110] Reason 2: During the process of other clients writing target data, one of these clients fails, resulting in at least k+1 shards out of M shards not being written to storage cluster 200. To address this situation, client 100 can possess feature 3.

[0111] Reason 3: After other clients write M shards to storage cluster 200, at least k+1 of these M shards may be corrupted or lost. To address this situation, client 100 can possess feature 3.

[0112] Feature 3: Triggers version rollback of target data.

[0113] As explained above, the target data stored in storage cluster 200 can be updated multiple times. To distinguish the target data after each update, the concept of a "version" is introduced. Each update to the target data creates a version of that data, and a version number is assigned to the updated data to indicate its version. In other words, the version number of the target data changes before and after an update. The existence of the version number distinguishes whether the target data is "new" or "old." "New" or "old" is defined based on the order in which the target data was updated. For example, the most recently updated target data is the "newest" version, while target data that has not been updated is the "old" version.

[0114] When processing a data read request, the m shards that client 100 needs to obtain through interaction with storage nodes 210 in storage cluster 200 should be m shards of the latest version of the target data. If client 100 fails to obtain m shards of the latest version of the target data during interaction with nodes in storage cluster 200, then client 100 can instruct storage cluster 200 to perform version rollback, reverting the current version of the target data to the previous version, and then interact with storage nodes 210 in storage cluster 200 again to obtain m shards of the previous version of the target data.

[0115] This application does not limit the specific form of the client 100. The client 100 can be a hardware module, such as a mobile terminal, a server, or other computing device, or a hardware component such as a processor. The client 100 can also be a software module, such as an application running on a computing device, or a computing instance such as a virtual machine or container deployed on a computing device.

[0116] The storage cluster 200 and the client 100 can be connected via a network. The storage cluster 200 has storage capabilities and includes multiple storage nodes 210 to support persistent data storage. The storage nodes 210 in the storage cluster 200 refer to nodes with storage capabilities. This application embodiment does not limit the specific form of the storage node 210; it can be a computing device with storage capabilities, a storage array, or a hard disk or other storage device.

[0117] In this embodiment of the application, any storage node 210 in the storage cluster 200 can, upon request from the client 100, return a fragment of the target data to the client 100, and can also, upon request from the client 100, store a fragment of the target data.

[0118] The following are some deployment methods for client 100 and storage cluster 200 in a data access system.

[0119] Deployment Method 1, such as Figure 3A As shown, an access device 300 is deployed on the user side, and an application and a client 100 are deployed on the access device 300. The application can interact with the direct user, generate data access requests upon user triggering, and send the data access requests to the client 100. This embodiment does not limit the specific type of the application; for example, the application could be a database management program, or it could be a file system management program, etc.

[0120] Client 100 can be viewed as the front-end software of storage cluster 200 deployed on the user side. It receives and processes data access requests from applications. In the process of processing data access requests from applications, client 100 interacts with storage cluster 200 to access the target data. The role of client 100 can be found in the preceding content and will not be repeated here.

[0121] Storage cluster 200 exists in the form of a distributed storage system, such as Figure 3A As shown, the distributed storage system includes at least one metadata server 230 and multiple data servers 230. The metadata server 230 is used to manage metadata. Metadata, also known as intermediary data or relay data, is data about data, mainly describing data properties, such as the data's storage address, modification history, size, and creation date.

[0122] Addressing client 100, the metadata server 230 can provide the client 100 with metadata of the target data. When client 100 receives a data write request, the data write request may only carry the identifier of the target data (such as the file name or object identifier of an object). Client 100 can initiate a query request to the metadata server 230 to query the storage address of the target data. The query request carries the identifier of the target data. After receiving the query request, the metadata server 230 can return the storage address of the target data to client 100.

[0123] After obtaining the storage address of the target data, the client 100 sends a data write instruction to M storage nodes 210 in the storage cluster 200. The data write instruction is used to request the writing of a fragment of the target data, and the data write instruction carries the storage address of the target data.

[0124] When client 100 receives a data read request, the data read request may only carry the identifier of the target data (such as the file name or the object identifier of the object). Client 100 can obtain the storage address of the target data from metadata server 230 through a query request. The process of client 100 obtaining the storage address of the target data from metadata server 230 through a query request can be found in the foregoing description, and will not be repeated here.

[0125] After obtaining the storage address of the target data, the client 100 sends a data read instruction to M storage nodes 210 in the storage cluster 200. The data read instruction is used to request to read a fragment of the target data, and the data read instruction carries the storage address of the target data.

[0126] The multiple data servers 230 are used to store and manage the stored data. Each data server 230 corresponds to the storage node 210 in the aforementioned data access system. The data server 230 can write and read data upon request from the client 100.

[0127] Deployment Method Two, such as Figure 3B The diagram shown is a schematic representation of a distributed system architecture provided in an embodiment of this application. The distributed system includes a server cluster. The server cluster includes one or more servers 250 ( Figure 3B The diagram shows three servers 250 (but is not limited to two servers 250), and these servers 250 can communicate with each other. A server 250 is a device that has both computing and storage capabilities, such as a server or desktop computer.

[0128] The aforementioned client 100 and storage node 210 are deployed on server 250. That is, server 250 has both the functions of client 100, which can receive and process data access requests from the outside, and the functions of storage node 210, which can store data.

[0129] When a data access request is used to request the writing of target data, server 250 generates M fragments of the target data (including m data fragments and k checksum fragments). Server 250 can save one of these fragments and store the remaining M-1 fragments on other servers. If server 250 sends a data write instruction to each of the M-1 servers 250, requesting the writing of one fragment of the target data, each of the M-1 servers 250, upon receiving the data write instruction, stores one fragment of the target data.

[0130] When a data access request is made to read target data, server 250 retrieves M fragments of the target data it has stored (including m data fragments and k verification fragments). Server 250 can store one fragment itself and retrieve m-1 fragments from other servers. If server 250 sends a data read instruction to some of the M-1 servers 250, requesting to read a fragment of the target data, any one of the M-1 servers 250, upon receiving the data read instruction, will return the fragment of the target data it has stored.

[0131] In terms of hardware, such as Figure 3B As shown, server 250 includes at least processor 251, memory 252, network interface card (NIC) 253, and hard disk 254. NIC 253 can be used to communicate with devices outside the server cluster, such as receiving data access requests from external sources. NIC 253 can also be used to communicate with hard disk 254.

[0132] Processor 251 can be a central processing unit (CPU) or other specific integrated circuits. Processor 251 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 251 is the main processing core within the server.

[0133] Memory 252 refers to the internal memory that directly exchanges data with processor 251. Memory includes at least one type of memory, such as random access memory (RAM) or read-only memory (ROM). For example, RAM can be dynamic random access memory (DRAM) or storage class memory (SCM). Memory 252 may also include other types of RAM, such as static random access memory (SRAM) or read-only memory. For read-only memory, it can be programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). Memory 252 may also be a dual in-line memory module (DIMM).

[0134] In this embodiment, memory 252 is used to temporarily store data, such as a copy of target data or a fragment of target data. Memory 252 may also store computer program instructions required by processor 251 to perform data operations.

[0135] The hard disk 254 is used to provide storage resources, such as storing data. It can be a disk or other types of storage media, such as a solid-state drive or a shingled magnetic recording hard disk. In this embodiment, the hard disk 254 can be used to persistently store fragments of target data.

[0136] Deployment method three, such as Figure 3C The diagram shown is a schematic diagram of the system architecture of another distributed system provided in this application embodiment. In this distributed system, the computing and storage functions required by the distributed storage system are deployed on different nodes. The node with computing function is called a computing node, and the node with storage function is called a storage node 210.

[0137] exist Figure 3C The distributed system comprises a computing cluster and a storage cluster 200. The computing cluster includes one or more computing nodes 110. Figure 3C The diagram shows two computing nodes 110 (but is not limited to two computing nodes 110), and the computing nodes 110 can communicate with each other. A computing node 110 is a computing device, such as a server, desktop computer, or controller of a storage array.

[0138] Compute node 110 is equipped with client 100, meaning it functions as a client. Compute node 110 also possesses computing capabilities, enabling it to receive and process data access requests from outside the system. During the processing of data access requests, compute node 110 can interact with various storage nodes 210 in the storage cluster 200 to access the target data.

[0139] In terms of hardware, such as Figure 3C As shown, computing node 110 includes at least processor 111, memory 112, and network interface card 113. The functions and specific types of processor 111, memory 112, and network interface card 113 are similar to those of processor 132, memory 133, and network interface card 134, and can be found in the foregoing content for details, which will not be repeated here.

[0140] Any compute node 110 can access any storage node 210 in the storage cluster via the network. The storage cluster includes multiple storage nodes 210 ( Figure 3C The diagram shows three storage nodes 210, but is not limited to three storage nodes 210. One storage node 210 includes one or more control units 211, a network interface card (NIC) 212, and multiple hard disks 213. The NIC 212 is used to communicate with the compute node 110. The hard disks 213 are used to store data and can be disks or other types of storage media, such as solid-state drives (SSDs) or shingled magnetic recording hard disks (SMARDs).

[0141] The functions of the control unit 211 (equivalent to the processor in storage node 210), network card 212 and multiple hard disks 213 are similar to those of the processor 111, network card 113 and hard disk 105 mentioned above. For details, please refer to the above description and it will not be repeated here.

[0142] The following is combined with Figure 4 This application provides a data access method according to an embodiment, such as... Figure 4 The diagram illustrates a data access method provided in an embodiment of this application. This data access method comprises two parts: a writing process for the target data, in which the client 100 uses an erasure coding verification mechanism to distribute M fragments of the target data across M storage nodes 210. See steps 401 to 406 for details. The other part is a reading process for the target data, in which the client 100 can use multiple rounds of interaction to retrieve the m fragments of the target data from the storage cluster 200. See steps 407 to 411 for details.

[0143] Step 401: Client 100 receives a data write request, which requests the writing of target data. The data write request carries the target data; optionally, it also carries a timestamp indicating the time the data write request was initiated. This embodiment does not limit the scenario in which client 100 receives the data write request; several scenarios are listed below:

[0144] Scenario 1: Client 100 receives a data write request sent by the application.

[0145] This application can interact directly with the user. It can respond to user actions, generate request messages, and send these messages to client 100. Because this application is closer to the user, the data processing operations performed by client 100 (such as handling data write requests) are triggered by the request messages sent by this application. Therefore, this application is located "above layer" of client 100 and is a higher-level application of client 100.

[0146] When a user needs to store target data, they can interact with the application to trigger the application to generate a data write request. Client 100 provides a data write interface for this application, which the application calls to send the data write request to client 100.

[0147] It should be noted that this data write interface represents a function provided by client 100 to the application; in other words, client 100 provides data write functionality to the application. This application embodiment does not limit the specific form of the data write interface. For example, the data write interface can be represented as a standard format for a data write request; that is, if the request message sent by the application conforms to this standard format, the request message can be considered a data write request. Alternatively, the data write interface can be represented as a function that the application can call.

[0148] Scenario 1 applies to the aforementioned deployment method 1, where the application and client 100 are deployed on the same computing device, and client 100 can interact with storage cluster 200 via the network. Of course, Scenario 1 also applies to the aforementioned deployment methods 2 and 3, where the application and client 100 are deployed on the same server 250 or computing node 111.

[0149] Scenario 2: Client 100 receives a data write request from a computing device deployed on the user side.

[0150] Scenario 2 is similar to Scenario 1, except that in Scenario 1, the sender of the data write request is an application, and the interaction between client 100 and the application is a software-level interaction. In Scenario 2, the sender of the data write request is a computing device, and client 100 interacts with this computing device.

[0151] This scenario applies to both deployment method 2 and deployment method 3 mentioned above, where client 100 is deployed on a computing node or server, and client 100 receives data write requests from computing devices deployed on the user side.

[0152] Scenario 3: Client 100 receives a data write request triggered by the user.

[0153] In some scenarios, client 100 can interact directly with the user. For example, client 100 provides a data writing interface to the user. The meaning of the data writing interface can be found in the preceding explanation. It's important to note that, unlike the previously mentioned data writing interfaces, this data writing interface is provided by client 100 to the user. This user-facing data writing interface can be represented as a visual interface, where the user can perform operations. Client 100 responds to the user's operations and receives data writing requests. Alternatively, the user-facing data writing interface can be represented as a command format. The user can transmit the data writing request to client 100 by typing commands that conform to this format. The commands typed by the user can be considered the data writing request.

[0154] It should be noted that the data write request can also carry other information, such as the storage address of the target data, indicating that client 100 needs to write the target data to that storage address. In some scenarios, the data write request may not carry the storage address of the target data, but instead carry the identifier of the target data. This identifier can be the identifier of the data set to which the target data belongs and the position of the target data within that data set. The specific form of this data set is related to the organization of data in storage cluster 200. For example, if storage cluster 200 has a file system deployed and data is organized in the form of files, then the data set is a file. The data write request can carry the filename of the file to which the target data belongs, and also the position of the target data within that file (such as the length of the target data and its offset within the file). In this case, after receiving the data write request, client 100 can first obtain the storage address of the target data, for example, by interacting with storage cluster 200 to request the storage address of the target data. Then, it can send a data write command to storage cluster 200, i.e., execute step 403.

[0155] Step 402: Client 100 caches a copy of the target data and obtains M fragments of the target data based on the target data.

[0156] After receiving the data write request, client 100 performs the following two operations:

[0157] Operation 1: Cache a copy of the target data.

[0158] In this embodiment, a cache space is deployed on the client 100 side, which can be the memory space of the computing device where the client 100 is located. After receiving a data write request, the client 100 caches a copy of the target data in the cache space.

[0159] Operation 2: Obtain M fragments of the target data.

[0160] Client 100 uses erasure coding to obtain M fragments of the target data. These M fragments include m data fragments and k check fragments. m and k are positive integers, and their specific values ​​can be set according to actual needs.

[0161] In some application scenarios, the data write request received by client 100 requests to write a large amount of data, or the size of the data is not suitable for calling erasure coding to generate fragments (usually, the amount of data that the erasure coding algorithm is designed for is fixed). Assume the data write request is for writing first data, which includes the target data.

[0162] When client 100 executes step 402, it divides the first data into multiple blocks, generating multiple data blocks. For any given data block, client 100 caches a copy of that data block and retrieves the data block based on it. That is, each data block can be considered as target data. The following explanation only uses the writing of target data as an example.

[0163] Step 403: Client 100 sends a data write instruction to M storage nodes 210. This data write instruction requests the writing of a fragment of the target data. The data write instruction carries a fragment of the target data, and the fragment carried in the data write instruction sent by client 100 to different storage nodes 210 is different. Optionally, the data write instruction carries a timestamp, which is consistent with the timestamp carried in the data write request. The data write instruction may also carry the storage address of the target data, instructing storage nodes 210 to write the fragment of the target data to that storage address.

[0164] There are multiple storage nodes 210 in the storage cluster 200. A "fault-tolerant group" is introduced in the storage cluster 200. The storage cluster 200 includes multiple fault-tolerant groups, and each fault-tolerant group includes M storage nodes 210.

[0165] When executing step 403, client 100 can determine the target fault-tolerant group from multiple fault-tolerant groups and send data write instructions to M storage nodes 210 in the target fault-tolerant group.

[0166] There are many ways for client 100 to determine the target fault-tolerant group from multiple fault-tolerant groups. For example, client 100 can use a round-robin rule to select one fault-tolerant group as the target fault-tolerant group. The round-robin rule means that client 100 selects each fault-tolerant group sequentially according to the order of the multiple fault-tolerant groups. Another example is that client 100 determines the target fault-tolerant group from multiple fault-tolerant groups based on the information of the target data. In addition to carrying the target data, the data write request can also carry information about the target data, such as the identifier of the target data or the storage address of the target data. Client 100 can hash the information of the target data to obtain a hash value. This hash value points to one of the multiple fault-tolerant groups, and the fault-tolerant group pointed to by the hash value is the target fault-tolerant group. This application embodiment does not limit the method by which client 100 determines the target fault-tolerant group from multiple fault-tolerant groups.

[0167] In step 403, client 100 requests M storage nodes 210 to store the M shards, wherein client 100 requests one storage node 210 to store one shard.

[0168] Step 404: Any one of the M storage nodes 210 receives a data write instruction and stores a fragment of the target data according to the data write instruction. After storing the fragment, a write success response is sent to the client 100 to indicate that the fragment was written successfully.

[0169] Since the target data is allowed to be updated multiple times in this data access system, the target data may change multiple times. In order to mark the different updates performed on the target data, storage node 210 sets a version number for the updated target data (or the fragment of the target data) each time the target data is updated.

[0170] Taking a fragment of target data as an example, after storing the fragment of target data, the storage node 210 also records the version number of the fragment. This embodiment does not limit the method of setting the version number. For example, the storage node 210 uses the timestamp carried in the data write instruction as the version number of the fragment. Alternatively, the storage node 210 may have a preset initial value for the version number. For newly written fragments, the version number of the fragment is set to this initial value. Each subsequent update of the fragment will result in the version number of the updated fragment being equal to the version number of the fragment before the update plus one. When executing step 404, if the target data is being written for the first time (e.g., the storage address of the target data is not occupied), the storage node 210 stores a fragment of the target data and records the version number of the fragment, which is the initial value. If the target data is not being written (e.g., the storage address of the target data is occupied), the storage node 210 stores a fragment of the target data and records the version number of the fragment, which is equal to the previously recorded version number of the fragment plus one.

[0171] When the storage node 210 stores the target data in a shard, it can store the shard in a log structure. That is, in addition to storing the shard, the storage node 210 will also store the log information of the shard. The log information describes the write operation of the shard, such as the time of shard writing, the storage address of the shard, the client 100 that requested to write to the shard, the version number of the shard, and the timestamp carried in the data write instruction.

[0172] Using a log structure to store the shard supports append-only writing of data. That is, if the shard needs to be updated later, the storage node 210 can continue to save the shard before the update and save the shard after the update, and add information related to this update to the log information, such as the update time of the shard, the storage address of the shard after the update, the version number of the shard after the update, and the client 100 that requested the update of the shard.

[0173] Because the storage node 210 uses a log structure for storage sharding, data can be recovered based on the recorded log information, meaning that data can be rolled back to any version.

[0174] Step 405: When the client 100 receives a write success response from m of the M storage nodes 210, it sends a data write response indicating that the target data has been written.

[0175] The successful write response from m out of the M storage nodes 210 indicates that m shards have been successfully written to the storage cluster 200. In this case, the client 100 does not need to wait for all M storage nodes 210 to send a successful write response, but sends a data write response to the sender of the data write response (such as an application, computing device, or user).

[0176] Since m shards have been successfully written, assuming k shards have not yet been successfully written, even if client 100 or other clients need to read the target data from storage cluster 200, storage cluster 200 can still provide the m shards to client 100 or other clients to ensure that client 100 or other clients can read the target data. Even if storage cluster 200 cannot provide the m shards to client 100 or other clients (if storage node 210 fails), client 100 still caches a copy of the target data. When client 100 needs to read the target data, it can obtain the copy, and other clients can interact with client 100 to obtain copies of the target data. Therefore, caching the target data copy and providing a write response after the m shards are written ensure both the efficiency and reliability of the target data.

[0177] Step 406: After receiving a write success response from the remaining k storage nodes 210 out of the M storage nodes 210, the client 100 sends a data write response again, which indicates that the target data has been written and deletes the cached copy of the target data.

[0178] After receiving write success responses from the m storage nodes 210, client 100 can continue to monitor the write status of the shards in the remaining k storage nodes 210, that is, continue to wait to receive write success responses from the remaining k storage nodes 210. Upon receiving write success responses from the remaining k storage nodes 210, client 100 will then send a data write response to the sender of the data write request again.

[0179] In other words, the sender of the data write request will receive two data write responses. Although both responses indicate that the target data has been written, they have different meanings. The first received data write response indicates that the target data has been written, meaning that the client 100 can support reading the target data, but the target data is not yet fully stored in the storage cluster 200. In other words, the first received response only indicates that the target data has been stored in the data access system. The second received data write response indicates that the target data has been written, meaning that the client 100 can support reading the target data, and the target data has been stored in the storage cluster 200. The second response represents the current actual storage status of the target data (i.e., the target data has been stored in the storage cluster 200).

[0180] The successful write responses from the remaining k storage nodes 210 indicate that all M shards have been stored in the storage cluster 200, achieving a redundancy of k. In this case, the cached copy of the target data on client 100 can be deleted. Client 100 can then read the target data from storage cluster 200.

[0181] Furthermore, in this embodiment, the client 100 provides a callback interface to external entities (such as users or applications). This callback interface is used to block processing requests for the target data following a data write request. The callback interface stops processing requests for the target data following the data write request after the client 100 has not responded with a second data write response.

[0182] Taking an application as an example, in practical applications, after the application requests the client 100 to write the target data via a data write request, it will continue to process the target data, such as compressing the target data, extracting key information from the target data, setting access permissions for the target data, etc. In other words, after sending the data write request, the application will also send a data processing request to request the processing of the target data.

[0183] According to the conventional processing flow, after client 100 initially responds with a data write response, the data processing request can begin to be processed, and the target data can be processed according to the data processing request (such as reading the target data and processing the target data). In this case, if some or all of the remaining k storage nodes 210 fail to respond with a successful write response, the processing of the target data by client 100 is often ineffective, especially when it involves modifications to the target data itself or its attributes (such as compressing the target data or setting access permissions for the target data). The processing of the target data cannot be fully synchronized to the storage cluster 200. Taking setting access permissions for the target data as an example, since there are unsuccessfully written shards on the M storage nodes 210, the access permissions for those shards cannot be recorded in the storage node 210.

[0184] Therefore, client 100 provides a callback interface. When the sender of a data write request calls the data write interface, it can also call this callback interface to instruct client 100 to process subsequent data processing requests only after the second data write response is received. That is, if the second data write response is not detected, subsequent data processing requests are "blocked" (i.e., processing of subsequent data processing requests is suspended). Similar to the data write interface, this application embodiment does not limit the specific form of the callback interface.

[0185] The following describes the process of client 100 interacting with storage cluster 200 to obtain target data. It is worth noting that multiple clients 100 can be deployed in this data access system. Therefore, during the process of client 100 interacting with storage cluster 200 to obtain target data, other clients are allowed to write to the target data (i.e., update the target data).

[0186] Step 407: Client 100 receives a data read request, which requests to read target data. This data read request may also carry other information, such as a timestamp indicating the time the data read request was initiated, information about the target data, and the storage address of the target data. For information about the storage address of the target data, please refer to the aforementioned explanation regarding data write requests. If the data read request does not include the storage address of the target data, client 100 can interact with storage node 210 to obtain the storage address of the target data.

[0187] The scenario in which client 100 receives a data read request is similar to the scenario in which client 100 receives a data write request, as described above. For details, please refer to the previous description, which will not be repeated here.

[0188] Step 408: Client 100 and storage cluster 200 perform one or more rounds of interaction to obtain m shards of the target data.

[0189] After receiving the data read request, client 100 determines the target fault-tolerant group and interacts with the storage node 210 in the target fault-tolerant group in multiple rounds to obtain m fragments of the target data. The purpose of each interaction between client 100 and the target fault-tolerant group is to obtain m fragments of the target data.

[0190] The method by which client 100 determines the target fault-tolerant group is similar to that in step 403. For example, if the target fault-tolerant group is determined using a polling rule in step 403, client 100 can record the correspondence between the target data (such as the target data information or the target data storage address) and the target fault-tolerant group. When executing step 408, client 100 determines the target fault-tolerant group based on this correspondence. Alternatively, if the target fault-tolerant group is determined based on the target data in step 403, client 100 can determine the target fault-tolerant group in the same way when executing step 408.

[0191] In each round of interaction, client 100 can interact with all storage nodes 210 in the target fault-tolerant group to obtain the m shards, or it can interact with only some storage nodes 210 in the target fault-tolerant group (such as the m storage nodes 210, or storage nodes 210 that were not interacted with in the previous round) to obtain the m shards. This application embodiment does not limit the specific implementation method of the interaction between client 100 and storage cluster 200 in each round; any method that can obtain m shards through interaction with storage cluster 200 is applicable to this application embodiment.

[0192] The following describes a method for client 100 to perform multiple rounds of interaction with storage cluster 200. The specific process is as follows:

[0193] First round of interaction:

[0194] Client 100 selects m storage nodes 210 from the target fault-tolerant group and sends a data read instruction to the m storage nodes 210. The data read instruction requests to read a fragment of the target data. The data read instruction may carry the storage address of the target data.

[0195] Upon receiving the data read / write instruction, any one of the m storage nodes 210 retrieves a fragment of the target data according to the data read instruction and sends the fragment back to the client 100. The storage node 210 can also send back the fragment's version number and the client 100 that requested to write to the fragment, i.e., information about the client 100 that requested to write to the fragment. This information can be the client 100's identifier or address information.

[0196] If client 100 successfully receives m shards from m storage nodes 210, it uses these m shards to obtain the target data. If storage nodes 210 also provide shard version numbers, client 100 first determines whether the version numbers of the m shards are consistent before obtaining the target data. If they are consistent, the client uses these m shards to obtain the target data; otherwise, it executes a second round of interaction.

[0197] If client 100 does not receive m shards from m storage nodes 210, that is, if there are storage nodes 210 among the m storage nodes 210 that have not returned shards, then the second round of interaction is executed.

[0198] Second round of interaction:

[0199] Client 100 sends a data read instruction to the remaining k storage nodes 210 in the target fault-tolerant group. This data read instruction requests to read a fragment of the target data. The data read instruction may carry the storage address of the target data.

[0200] Storage node 210 handles the data read command in a similar manner, as detailed in the preceding description, and will not be repeated here.

[0201] Client 100 receives fragmentation data from some or all of the k storage nodes 210.

[0202] Here, it is assumed that client 100 interacts with k storage nodes 210 to obtain p shards, where p is a positive integer not greater than k. In the first round of interaction, client 100 interacts with m storage nodes 210 to obtain q shards, where q is a positive integer not greater than m.

[0203] Client 100 selects m shards from the p shards and q shards, and uses the m shards to obtain the target data. If the storage node 210 also provides the version number of the shards, client 100 selects the m shards with the same version number when selecting the m shards, and uses the m shards with the same version number to obtain the target data.

[0204] If client 100 cannot select m shards from the p shards and q shards, that is, if there are no m shards with the same version number among the p shards and q shards, then step 409 can be executed.

[0205] Step 409: If client 100 does not obtain m shards of the target data from storage cluster 200, client 100 obtains a copy of the target data from other clients.

[0206] If client 100 fails to retrieve m shards of the target data from storage cluster 200, it indicates that the shards of the target data may not yet be fully stored in the target fault-tolerant group. In other words, while client 100 is reading the target data, other clients are interacting with storage cluster 200 to write the target data, and these other clients have not yet stored all shards of the target data in storage cluster 200. In this case, these other clients will have a cached copy of the target data.

[0207] Since storage node 210 also provides information about other clients requesting to write the target data fragments to client 100 when it sends the fragments of the target data back to the client 100, the client 100 can identify these other clients based on this information and obtain a copy of the target data from them.

[0208] Step 410: If client 100 does not obtain a copy of the target data from other clients, client 100 instructs storage cluster 200 to perform a version rollback of the target data and obtains the previous version of the target data from storage cluster 200.

[0209] There are many reasons why client 100 may not have obtained a copy of the target data from other clients. For example, other clients may be malfunctioning; or other clients may have deleted a copy of the target data.

[0210] That is, if there are not m shards with the same version number stored in the target fault-tolerant group, then the client 100 can instruct the storage cluster 200 to perform a version rollback.

[0211] This application embodiment does not limit the method of version rollback within the storage cluster 200. For example, the client 100 can instruct a storage node 210 in the target fault tolerance group to perform version rollback for the target data. The storage node 210 interacts with other storage nodes 210 in the target fault tolerance group to negotiate the version number to be rolled back. After that, each storage node 210 in the target fault tolerance group rolls back its saved allocation to that version (that is, the version represented by the negotiated version number).

[0212] After performing the version rollback of the target data, storage cluster 200 can notify client 100 that the rollback was successful, and client 100 can continue to interact with storage cluster 200 to obtain the target data. The method by which client 100 interacts with storage node 210 to obtain the target data can be found in the relevant description of step 408 above, and will not be repeated here.

[0213] Step 411: Client 100 sends a data read response, which carries the target data to be read.

[0214] Client 100 can send the data read response to the sender of the data read request to provide the target data to the sender of the data read request.

[0215] Based on the same inventive concept as the method embodiments, this application also provides a data access device 500, which is used to execute the method executed by the client 100 in the above method embodiments. For example... Figure 5 As shown, the main device 500 includes a receiving module 501, a processing module 502, and a sending module 503. Specifically, in the data access device 500, the modules are connected through a communication path.

[0216] The receiving module 501 is used to receive data write requests, which are used to request the writing of target data.

[0217] Processing module 502 is used to cache copies of the target data and call erasure coding to obtain M fragments of the target data, where M is a positive integer.

[0218] The sending module 503 is used to instruct the storage cluster to store M shards. When it is determined that m shards out of the M shards have been stored in the storage cluster, it sends a data write response. The data write response indicates that the target data has been successfully written, where m is less than M and m is a positive integer.

[0219] As one possible implementation, if the processing module 502 determines that all M shards are stored in the storage cluster, it deletes the cached copy of the target data and sends back a data write response.

[0220] In one possible implementation, the receiving module 501 receives a data read request, which requests to read target data. The processing module 502 obtains a copy of the target data from another client. The sending module 503 sends a data read response, which carries a copy of the target data.

[0221] As one possible implementation, the processing module 502 obtains m shards of the target data from the storage cluster.

[0222] As one possible implementation, a data write request is used to request the writing of first data, which includes multiple target data. When m fragments of each target data have been stored in the storage cluster, the sending module 503 sends a data write response, which indicates that the first data has been successfully written.

[0223] As one possible implementation, the storage cluster includes multiple fault-tolerant groups, each of which includes M storage nodes. The sending module 503 determines the target fault-tolerant group from the multiple fault-tolerant groups and sends data write instructions to the M storage nodes in the target fault-tolerant group respectively. The data write instructions are used to request the writing of a fragment of the target data. The data write instructions sent to different storage nodes request the writing of different fragments.

[0224] In one possible implementation, the sending module 503 receives write success responses from m storage nodes in the target fault-tolerant group. The write success responses are used to indicate that the fragment write was successful.

[0225] As one possible implementation, the processing module 502 interacts with the target fault-tolerant group at least once, but fails to acquire m of the M fragments.

[0226] In one possible implementation, the processing module 502 obtains information about other clients from the target fault-tolerant group, where the information about other clients is their identifiers or address information; and obtains copies of the target data from the other clients based on the information about other clients.

[0227] As one possible implementation, the processing module 502 interacts with the target fault-tolerant group at least once, and does not acquire m fragments with the same version number as the target data.

[0228] The module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0229] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0230] This application also provides, for example Figure 6 The computing device 600 shown includes a bus 601, a processor 602, a communication interface 603, and a memory 604. The processor 602, the memory 604, and the communication interface 603 communicate with each other via the bus 601.

[0231] The processor 602 can be a CPU or other specific integrated circuits. The processor 62 can also be other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0232] Memory 604 can be DRAM. Besides DRAM, memory 604 can also be other random access memory (such as SRAM). Additionally, memory 602 can also be ROM. For read-only memory, for example, it could be PROM, EPROM, etc. Memory 604 can also be flash memory, HDD, or SSD.

[0233] The memory 604 stores computer program instructions, and the processor 602 executes these computer program instructions to perform the aforementioned tasks. Figure 4 The steps executed by client 100. The memory 604 may also include software modules required for other running processes, such as the operating system (e.g., multiple modules in the data access device 500). The operating system may be LINUX. TM UNIX TM WINDOWS TM wait.

[0234] This application also provides a computing device system, the computing device system including at least one such as Figure 7 The computing device 700 shown includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, memory 704, and communication interface 703 communicate with each other via the bus 701. At least one computing device 700 in the computing device system communicates with each other via a communication path.

[0235] The specific types of processor 702 and memory 704 can be found in the relevant descriptions of processor 602 and memory 604, and will not be repeated here. Processor 702 executes the computer program instructions stored in memory 704 to perform the aforementioned tasks. Figure 4 The described method includes some or all of the steps executed by client 100. The memory may also include other software modules required for running processes, such as the operating system. The operating system may be Linux. TM UNIXTM WINDOWS TM wait.

[0236] At least one computing device 700 in the computing device system establishes communication with each other through a communication network, and each computing device 700 runs any one or any multiple modules of the data access device 500.

[0237] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0238] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, in the form of a computer program product. A computer program product includes computer program instructions, which, when loaded and executed on a computer, generate, in whole or in part, the product according to the embodiments of the present invention. Figure 5 The process or function described.

[0239] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD).

[0240] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data access system, characterized in that, The system includes: The first client is configured to receive a data write request, the data write request being used to request the writing of target data; cache a copy of the target data, and invoke erasure coding to obtain M fragments of the target data; instruct the storage cluster to store the M fragments, where M is a positive integer; The storage cluster is used to: store the M shards as instructed by the first client; The first client is further configured to: when it is determined that m out of the M shards have been stored in the storage cluster, provide a data write response, wherein the data write response indicates that the target data has been successfully written, and m is less than M and m is a positive integer.

2. The system as described in claim 1, characterized in that, The first client is also used for: If it is determined that all M shards are stored in the storage cluster, delete the cached copy of the target data and send back the data write response again.

3. The system as described in claim 1, characterized in that, The system also includes a second client, which is used for: Receive a data read request, the data read request being used to request the reading of the target data; Obtain a copy of the target data from the first client; Feedback data read response, the data read response carrying a copy of the target data.

4. The system as described in claim 3, characterized in that, Before obtaining a copy of the target data from the first client, the second client is further configured to: m shards of the target data were not retrieved from the storage cluster.

5. The system according to any one of claims 1 to 4, characterized in that, The data write request is used to request the writing of first data, the first data including multiple target data, and the first client is used for: When m shards of each target data have been stored in the storage cluster, a data write response is fed back, indicating that the first data has been successfully written.

6. The system according to any one of claims 1 to 5, characterized in that, The storage cluster includes multiple fault-tolerant groups, each fault-tolerant group includes M storage nodes, and the first client is used for: Determine the target fault-tolerant group from the plurality of fault-tolerant groups; Data write instructions are sent to M storage nodes in the target fault-tolerant group respectively. The data write instructions are used to request the writing of a fragment of the target data. The data write instructions sent to different storage nodes request the writing of different fragments.

7. The system as described in claim 6, characterized in that, The first client is used for: Upon receiving write success responses from m storage nodes in the target fault-tolerant group, the write success responses are used to indicate that the shard write was successful.

8. The system as described in claim 6, characterized in that, The second client is used for: After at least one round of interaction with the target fault-tolerant group, m of the M fragments were not acquired.

9. The system as described in claim 6, characterized in that, The second client is used for: The information of the first client is obtained from the target fault-tolerant group, wherein the information of the first client is the identifier or address information of the first client; A copy of the target data is obtained from the first client based on the information from the first client.

10. The system according to any one of claims 6 to 9, characterized in that, Any storage node in the target fault-tolerant group is configured to: receive the data write instruction, store a shard of the target data according to the data write instruction, and record log information of the target data, wherein the log information includes the version number of the shard and information of the first client requesting to write to the shard.

11. A data access method, characterized in that, The method includes: The first client receives a data write request, which is used to request the writing of target data. The first client caches a copy of the target data and invokes erasure coding to obtain M fragments of the target data, where M is a positive integer; The first client instructs the storage cluster to store the M shards. When it is determined that m shards out of the M shards have been stored in the storage cluster, it sends a data write response, which indicates that the target data has been successfully written. Here, m is less than M and m is a positive integer.

12. The method as described in claim 11, characterized in that, The method further includes: Once the first client determines that all M shards are stored in the storage cluster, it deletes the cached copy of the target data and sends back the data write response again.

13. The method as described in claim 11, characterized in that, The method further includes: The second client receives a data read request, which is used to request the reading of the target data; The second client obtains a copy of the target data from the first client; The second client sends a data read response, which carries a copy of the target data.

14. A data access method, characterized in that, The method is applied to a client, and the method includes: Receive a data write request, the data write request being used to request the writing of target data; Cache a copy of the target data and call erasure coding to obtain M fragments of the target data, where M is a positive integer; The system instructs the storage cluster to store the M shards. When it is determined that m shards out of the M shards have been stored in the storage cluster, a data write response is fed back, indicating that the target data has been successfully written. Here, m is less than M and m is a positive integer.

15. The method as described in claim 14, characterized in that, The method further includes: If it is determined that all M shards are stored in the storage cluster, delete the cached copy of the target data and send back the data write response again.

16. The method as described in claim 14, characterized in that, The method further includes: Receive a data read request, the data read request being used to request the reading of the target data; Obtain a copy of the target data from another client; Feedback data read response, the data read response carrying a copy of the target data.

17. A data access device, characterized in that, The device includes: The receiving module is used to receive data write requests, wherein the data write requests are used to request the writing of target data; The processing module is used to cache a copy of the target data and call erasure coding to obtain M fragments of the target data, where M is a positive integer; The sending module is used to instruct the storage cluster to store the M shards. When it is determined that m shards out of the M shards have been stored in the storage cluster, it sends a data write response, which indicates that the target data has been successfully written. Here, m is less than M and m is a positive integer.

18. The apparatus as claimed in claim 17, characterized in that, The processing module is further configured to: If it is determined that all M shards are stored in the storage cluster, delete the cached copy of the target data and send back the data write response again.

19. The apparatus as claimed in claim 17, characterized in that, The receiving module is further configured to: receive a data read request, wherein the data read request is used to request the reading of the target data; The processing module is further configured to: obtain a copy of the target data from other clients; The sending module is further configured to: provide a data reading response, wherein the data reading response carries a copy of the target data.

20. A computing device, characterized in that, The computing device includes a memory and a processor; The memory stores computer program instructions; The processor is used to invoke computer program instructions in the memory to execute the method as described in any one of claims 11 to 16.

Images