Data processing method and device, distributed storage system, storage medium and product

Abstract

The invention relates to a data processing method and device, a distributed storage system, a storage medium and a product. The method comprises the following steps: when a target cluster receives a processing request, determining a storage position of to-be-processed data according to metadata corresponding to the to-be-processed data requested by the processing request; updating first metadata corresponding to the to-be-processed data on the target cluster and second metadata corresponding to the to-be-processed data on other clusters in the distributed storage system according to the attribute of the to-be-processed data and the file residual capacity where the storage position is located; and processing the to-be-processed data according to the updated metadata corresponding to the to-be-processed data. By adopting the method, the metadata corresponding to the to-be-processed data on each cluster in the distributed storage system can be ensured to be consistent all the time, so that the situation that the data is difficult to read successfully on the bilateral cluster due to data inconsistency is avoided, and the data reading success rate of each cluster in the distributed storage system can be improved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a data processing method, apparatus, distributed storage system, storage medium and product. Background Technology

[0002] A distributed storage system consists of multiple server clusters, such as source server clusters and remote server clusters. During the operation of a distributed storage system, data in the source server cluster can be migrated or backed up to the remote server cluster to ensure the security of the data stored in the distributed storage system.

[0003] However, after completing the above data migration or backup, the source server cluster or remote server cluster has a low read success rate. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, distributed storage system, storage medium, and product that can improve the read success rate of source server clusters or remote server clusters, addressing the aforementioned technical problems.

[0005] In a first aspect, this application provides a data processing method applied to a distributed storage system, the distributed storage system including multiple clusters, the method comprising:

[0006] When the target cluster receives a processing request, it determines the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested in the processing request.

[0007] Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0008] The data to be processed is processed based on the updated metadata corresponding to the data to be processed.

[0009] In the above data processing method, the metadata corresponding to the data to be processed on the target cluster and other clusters can be updated synchronously. In this way, it can be ensured that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always consistent, thereby avoiding the difficulty of successfully reading data on both sides of the cluster due to data inconsistency. Therefore, the above method can improve the data reading success rate of each cluster in the distributed storage system.

[0010] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed. Updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, includes:

[0011] The cluster type to which the data to be processed belongs is determined based on the storage location, and the capacity of the data to be processed is compared with the remaining capacity of the file where the storage location is located;

[0012] If the cluster type is the target cluster, then according to the comparison result, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters are updated using the first update method.

[0013] If the cluster type is another cluster, then the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other cluster are updated using the second update method according to the comparison result.

[0014] In this embodiment, the cluster type to which the data to be processed belongs can be determined based on the storage location, and the capacity of the data to be processed can be compared with the remaining capacity of the file where the storage location is located. Thus, based on the comparison result and whether the cluster type is the target cluster or another cluster, the update method corresponding to the cluster type can be adopted to accurately update the metadata corresponding to the data to be processed on the target cluster and other clusters synchronously. In this way, it can be ensured that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state.

[0015] In one embodiment, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters using a first update method based on the comparison result includes:

[0016] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then a first data segment is allocated for the data to be processed on the target cluster, and the first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other clusters; the first update request message is used to instruct the other clusters to allocate a second data segment for the data to be processed, and to update the second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0017] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on the other clusters is updated according to the metadata corresponding to the data to be processed.

[0018] In this embodiment, it can be accurately determined whether a new data segment needs to be allocated for the data to be processed based on the comparison results. Thus, based on whether a new data segment needs to be allocated for the data to be processed and the metadata corresponding to the data to be processed, the first metadata corresponding to the data to be processed on the target cluster is updated first, and then the second metadata corresponding to the data to be processed on other clusters is updated. This ensures that when the cluster type is the target cluster, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is always in the latest state.

[0019] In one embodiment, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters using a second update method based on the comparison result includes:

[0020] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then a third data segment is allocated for the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment for the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0021] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0022] In this embodiment, it is possible to accurately determine whether a new data segment needs to be allocated to the data to be processed based on the comparison results. Thus, based on whether a new data segment needs to be allocated to the data to be processed and the metadata corresponding to the data to be processed, the second metadata corresponding to the data to be processed on other clusters is updated first, and then the first metadata corresponding to the data to be processed on the target cluster is updated. This ensures that when the cluster type is other clusters, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is always in the latest state.

[0023] In one embodiment, processing the data to be processed based on the updated metadata corresponding to the data to be processed includes:

[0024] The processing location is determined based on the updated metadata corresponding to the data to be processed;

[0025] If the processing request is a write request, then new data is written at the processing location;

[0026] If the processing request is a modification request, then the data to be processed is modified.

[0027] In this embodiment, the processing location can be accurately determined based on the updated metadata corresponding to the data to be processed. If the processing request is a write request, new data is written at the processing location; if the processing request is a modification request, the data to be processed is modified. In this way, the data to be processed can be processed adaptively according to different processing requests.

[0028] In one embodiment, the method further includes:

[0029] When the target cluster receives a read request, it determines the storage location of the target data based on the metadata corresponding to the target data requested by the read request.

[0030] Read the target data from the storage location of the target data.

[0031] In this embodiment, since the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is always up-to-date, when the target cluster receives a read request, it can accurately determine the storage location of the target data based on the accurate metadata corresponding to the target data requested by the read request. Thus, the target data can be accurately read from the storage location of the target data, thereby improving the data read success rate of each cluster in the distributed storage system.

[0032] Secondly, this application also provides a data processing apparatus, comprising:

[0033] The determination module is used to determine the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request when the target cluster receives a processing request.

[0034] The update module is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located.

[0035] The processing module is used to process the data to be processed according to the updated metadata corresponding to the data to be processed.

[0036] Thirdly, this application also provides a distributed storage system. The distributed storage system includes a source server cluster and a remote server cluster, and the source server cluster and the remote server cluster are connected in communication.

[0037] The source server cluster is used to determine the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request when the source server cluster receives a processing request.

[0038] Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the source server cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0039] The data to be processed is processed according to the updated metadata corresponding to the data to be processed;

[0040] Alternatively, the remote server cluster is used to determine the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request when the remote server cluster receives a processing request.

[0041] Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the remote server cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0042] The data to be processed is processed based on the updated metadata corresponding to the data to be processed.

[0043] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0045] The aforementioned data processing method, apparatus, distributed storage system, storage medium, and product, upon receiving a processing request from the target cluster, determine the storage location of the data to be processed based on the metadata corresponding to the requested data; update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located; and process the data to be processed based on the updated metadata. This embodiment of the application can synchronously update the metadata corresponding to the data to be processed on the target cluster and other clusters. This ensures that the metadata corresponding to the data to be processed on each cluster in the distributed storage system remains consistent, thereby avoiding difficulties in successfully reading data on both sides of the cluster due to data inconsistency. Therefore, the above method can improve the data read success rate of each cluster in the distributed storage system. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0048] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0049] Figure 3 This is a flowchart illustrating the metadata update steps in one embodiment;

[0050] Figure 4 This is a flowchart illustrating the data processing steps in one embodiment;

[0051] Figure 5 This is a flowchart illustrating the data processing method in another embodiment;

[0052] Figure 6 This is a flowchart illustrating the data processing method in one optional embodiment;

[0053] Figure 7 This is a schematic diagram illustrating the source cluster write service in one embodiment;

[0054] Figure 8 This is a flowchart illustrating a data processing method when the target cluster is the source cluster in one embodiment.

[0055] Figure 9 This is a schematic diagram of a remote cluster write service in one embodiment;

[0056] Figure 10 This is a flowchart illustrating a data processing method when the target cluster is a remote cluster, as shown in one embodiment.

[0057] Figure 11 This is a structural block diagram of a data processing device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0060] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0061] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] A distributed storage system consists of multiple server clusters, such as source server clusters and remote server clusters. During the operation of a distributed storage system, data in the source server cluster can be migrated or backed up to the remote server cluster to ensure the security of the data stored in the distributed storage system.

[0063] Traditional techniques for cross-cluster data storage write copies of the data to both the source and remote server clusters, achieving synchronized backup between them. This backup method requires two separate data storage spaces within the distributed storage system. However, since data in a file cannot be modified simultaneously on both clusters in the distributed storage system, any modification or write operation performed on either the source or target server cluster will result in discrepancies between the two clusters. Consequently, subsequent attempts to retrieve the same data from either cluster will fail. Therefore, traditional techniques suffer from low read success rates on either the source or remote server clusters after data migration or backup.

[0064] Having described the background technology of the data processing method provided in the embodiments of this application, the implementation environment involved in the data processing method provided in the embodiments of this application will be briefly described below. The data processing method provided in the embodiments of this application can be applied to, for example... Figure 1 The distributed storage system shown includes various types of server clusters. These different types of server clusters can communicate, transmit data, and exchange information with each other. Figure 1 The distributed storage system shown includes a source server cluster 101 and a remote server cluster 102. Both the source server cluster 101 and the remote server cluster 102 can perform bilateral business processing; for example, both the source server cluster and the remote server cluster support read and write operations. Each type of server cluster can be a cluster composed of multiple physical servers. These physical servers can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0065] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the distributed storage system on which the solution of this application is applied. A specific distributed storage system may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0066] In one embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking its application in a distributed storage system as an example, the following steps are included:

[0067] S201: When the target cluster receives a processing request, it determines the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested in the processing request.

[0068] In this context, the target cluster refers to any cluster in the distributed storage system. Processing requests can include, but are not limited to, write requests, modification requests, and other data processing requests. Each processing request carries data to be processed, which refers to the data within the cluster that the processing request requests to process. For example, if the processing request is a write request, the data to be processed refers to the data where new data needs to be written to the corresponding location in the cluster. The metadata corresponding to the data to be processed can include the location and capacity of the data. The storage location of the data to be processed can be in any cluster of the distributed storage system. That is, if multiple clusters include a source server cluster and a remote server cluster, the data to be processed can be stored in the source server cluster, or it can be stored in the remote server cluster, and both the source and remote server clusters store the same metadata corresponding to the data to be processed.

[0069] In this embodiment, when a target cluster in a distributed storage system receives a processing request, the target cluster can determine the metadata corresponding to the data to be processed based on the requested data. Therefore, the target cluster can determine the storage location of the data to be processed based on the location of the data to be processed included in the metadata. For example, the target cluster can determine whether the data to be processed is stored in the target cluster based on the metadata. If the metadata indicates that the data to be processed is stored in the target cluster, the target cluster can determine the storage location of the data to be processed in the target cluster based on the location of the data to be processed included in the metadata. If the metadata indicates that the data to be processed is stored in another cluster, the target cluster can send a location retrieval request to the other clusters. This instructs the other clusters, upon receiving the location retrieval request, to determine the storage location of the data to be processed on the other clusters based on the location of the data to be processed included in the metadata, and send the storage location of the data to the target cluster. Thus, the target cluster can receive the storage location of the data to be processed on the other clusters returned by the other clusters. Here, "other clusters" refers to clusters in the distributed storage system other than the target cluster.

[0070] S202, based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0071] The attributes of the data to be processed may include, but are not limited to, the capacity of the data to be processed and the amount of resources occupied by the data to be processed. The remaining capacity of the file where the storage location is located refers to the size of the remaining storage space corresponding to the file where the data to be processed is stored. The first metadata refers to the metadata corresponding to the data to be processed on the target cluster, and the second metadata refers to the metadata corresponding to the data to be processed on other clusters. It should be noted that the embodiments of this application can synchronously update the metadata corresponding to the data to be processed on each cluster in the distributed storage system, thus ensuring that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same.

[0072] In this embodiment, the target cluster can determine the attributes and remaining capacity of the file containing the storage location of the data to be processed based on the metadata corresponding to the data to be processed. Then, based on the attributes and remaining capacity of the file containing the storage location of the data to be processed, it determines whether a new data segment needs to be allocated for the data to be processed. Therefore, the target cluster can update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system based on the storage location of the data to be processed and whether a new data segment needs to be allocated for the data to be processed. Optionally, the target cluster can first update the first metadata corresponding to the data to be processed on the target cluster, and then update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system; or, the target cluster can first update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, and then update the first metadata corresponding to the data to be processed on the target cluster. Of course, this embodiment does not limit the order in which the first and second metadata are updated.

[0073] S203, Process the data to be processed based on the updated metadata corresponding to the data to be processed.

[0074] This application involves two methods for processing the data to be processed. The first method involves processing the data directly at its storage location based on the updated metadata, provided the metadata indicates the data is stored in the target cluster. The second method involves sending a processing request message to the other cluster, instructing them to process the data based on the updated metadata and send a processing response message back to the target cluster. The response message indicates that the processing of the data has been completed. Of course, this application does not limit the specific implementation method for processing the data.

[0075] In the above data processing method, when the target cluster receives a processing request, the storage location of the data to be processed is determined based on the metadata corresponding to the data to be processed requested in the processing request; the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system are updated based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located; and the data to be processed is processed based on the updated metadata corresponding to the data to be processed. This embodiment of the application can synchronously update the metadata corresponding to the data to be processed on the target cluster and other clusters. This ensures that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always consistent, thereby avoiding difficulties in successfully reading data on both sides of the cluster due to data inconsistency. Therefore, the above method can improve the data reading success rate of each cluster in the distributed storage system.

[0076] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed. Based on this, an implementation method for updating the first metadata and the second metadata is provided, namely, "updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attributes of the data to be processed and the remaining capacity of the file where the storage location is located" in S202 above. Figure 3 As shown, it includes:

[0077] S301 determines the cluster type of the data to be processed based on the storage location, and compares the capacity of the data to be processed with the remaining capacity of the file where the storage location is located.

[0078] Among them, the attributes of the data to be processed include the capacity of the data to be processed, which refers to the size of the storage space occupied by the data to be processed, and the cluster type can include the target cluster or other clusters.

[0079] In this embodiment, the target cluster can determine whether the cluster to which the data to be processed belongs is the target cluster or another cluster based on the storage location of the data to be processed. Furthermore, the target cluster can compare the capacity of the data to be processed with the remaining capacity of the file where the storage location is located to obtain a comparison result. The comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, or that the capacity of the data to be processed is not greater than the remaining capacity of the file. Of course, this embodiment does not limit the order in which the cluster type is determined and the comparison result is obtained.

[0080] S302, if the cluster type is the target cluster, then according to the comparison result, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters are updated using the first update method.

[0081] In this embodiment, if the cluster type to which the data to be processed belongs is determined to be the target cluster, the target cluster can determine whether a new data segment needs to be allocated for the data to be processed based on the comparison results. Therefore, depending on whether a new data segment needs to be allocated, the first update method is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters. The first update method is as follows: first update the first metadata corresponding to the data to be processed on the target cluster, and then update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system. That is, if the cluster type is the local cluster (i.e., the cluster receiving the processing request has the same cluster type as the data to be processed), the local cluster can first update the first metadata corresponding to the data to be processed on its own cluster, and then update the second metadata corresponding to the data to be processed on the other clusters (i.e., the peer cluster).

[0082] S303, if the cluster type is other clusters, then according to the comparison results, the second update method is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters.

[0083] In this embodiment, if it is determined that the cluster type to which the data to be processed belongs is another cluster, the target cluster can determine whether it needs to allocate a new data segment for the data to be processed based on the comparison results. Therefore, depending on whether a new data segment needs to be allocated, the second update method can be used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters. The second update method is as follows: first update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, and then update the first metadata corresponding to the data to be processed on the target cluster. That is, if the cluster type is the peer cluster (i.e., the cluster receiving the processing request is different from the cluster type to which the data to be processed belongs), the local cluster can first update the second metadata corresponding to the data to be processed on the peer cluster, and then update the first metadata corresponding to the data to be processed on the cluster other than the peer cluster (i.e., the local cluster).

[0084] In this embodiment, the cluster type to which the data to be processed belongs can be determined based on the storage location, and the capacity of the data to be processed can be compared with the remaining capacity of the file where the storage location is located. Thus, based on the comparison result and whether the cluster type is the target cluster or another cluster, the update method corresponding to the cluster type can be adopted to accurately update the metadata corresponding to the data to be processed on the target cluster and other clusters synchronously. In this way, it can be ensured that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state.

[0085] In one embodiment, an implementation method for updating the first metadata and the second metadata using a first update method is provided, namely, "updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters using the first update method according to the comparison result" in S302 above, including:

[0086] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining file capacity, then a first data segment is allocated for the data to be processed on the target cluster, and the first metadata corresponding to the data to be processed is updated on the first data segment; a first update request message is sent to other clusters; the first update request message is used to instruct other clusters to allocate a second data segment for the data to be processed, and to update the second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size. If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining file capacity, then the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed.

[0087] In this embodiment, the target cluster can compare the capacity of the data to be processed with the remaining capacity of the file where it is stored to obtain a comparison result. If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, it means that a new data segment needs to be allocated for the data to be processed. At this time, the target cluster can determine the size of the data segment to be allocated based on the difference between the capacity of the data to be processed and the remaining capacity of the file, and allocate a first data segment for the data to be processed on the target cluster, and update the first metadata corresponding to the data to be processed on the first data segment. After the first metadata update is completed, the target cluster can send a first update request message to other clusters, so that after receiving the first update request message, the other clusters can allocate a second data segment for the data to be processed based on the size of the first data segment, and update the second metadata corresponding to the data to be processed on the second data segment. The size of the data segment to be allocated, the size of the first data segment, and the size of the second data segment are all the same. The first update request message is used to instruct other clusters to allocate a second data segment for the data to be processed, and to update the second metadata corresponding to the data to be processed on the second data segment. A data segment refers to slicing the data in a file; each sliced ​​data storage unit can be considered a data segment.

[0088] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining file capacity, it means that no new data segment needs to be allocated for the data to be processed. In this case, the target cluster can determine the processing location and processing capacity corresponding to the processing request based on the metadata corresponding to the data to be processed, and thus update the first metadata on the target cluster according to the processing location and processing capacity. After the first metadata update is completed, a third update request message is sent to other clusters, so that the other clusters, upon receiving the third update request message, update the second metadata on their own clusters according to the metadata corresponding to the data to be processed. The third update request message instructs other clusters to update their second metadata according to the metadata corresponding to the data to be processed. It should be noted that the updated first metadata and the updated second metadata are the same.

[0089] In this embodiment, it can be accurately determined whether a new data segment needs to be allocated for the data to be processed based on the comparison results. Thus, based on whether a new data segment needs to be allocated for the data to be processed and the metadata corresponding to the data to be processed, the first metadata corresponding to the data to be processed on the target cluster is updated first, and then the second metadata corresponding to the data to be processed on other clusters is updated. This ensures that when the cluster type is the target cluster, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is always in the latest state.

[0090] In one embodiment, an implementation method for updating the first metadata and the second metadata using a second update method is provided, namely, "updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters using the second update method according to the comparison result" in S303 above, including:

[0091] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining file capacity, then a third data segment is allocated for the data to be processed on other clusters, and the second metadata corresponding to the data to be processed is updated on the third data segment; a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment for the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third and fourth data segments have the same size. If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining file capacity, then the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0092] In this embodiment, the target cluster can compare the capacity of the data to be processed with the remaining capacity of the file where it is stored to obtain a comparison result. If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, it means that a new data segment needs to be allocated for the data to be processed. At this time, the target cluster can determine the size of the data segment to be allocated based on the difference between the capacity of the data to be processed and the remaining capacity of the file, and send a fourth update request message to other clusters. After receiving the fourth update request message, other clusters can allocate a third data segment for the data to be processed on other clusters according to the size of the data segment to be allocated, and update the second metadata corresponding to the data to be processed on the third data segment. After the second metadata update is completed, other clusters can send a second update request message to the target cluster. After receiving the second update request message, the target cluster can allocate a fourth data segment for the data to be processed according to the size of the third data segment, and update the first metadata corresponding to the data to be processed on the fourth data segment. If the comparison result indicates that the size of the data to be processed is not greater than the remaining file size, it means that no new data segment needs to be allocated for the data to be processed. In this case, the target cluster can send a fifth update request message to other clusters, so that after receiving the fifth update request message, the other clusters can update the second metadata on their own clusters according to the metadata corresponding to the data to be processed. After the second metadata update is completed, the target cluster can determine the processing position and processing capacity corresponding to the processing request based on the metadata corresponding to the data to be processed, and then update the first metadata on its own cluster according to the processing position and processing capacity. The fifth update request message is used to instruct other clusters to update the second metadata on their own clusters according to the metadata corresponding to the data to be processed. It should be noted that the updated first metadata and the updated second metadata are the same.

[0093] The data segments to be allocated, the third data segment, and the fourth data segment all have the same size. The second update request message instructs the target cluster to allocate the fourth data segment for the data to be processed and to update the first metadata corresponding to the data to be processed on the fourth data segment. The fourth update request message instructs other clusters to allocate the third data segment for the data to be processed and to update the second metadata corresponding to the data to be processed on the third data segment.

[0094] In this embodiment, it is possible to accurately determine whether a new data segment needs to be allocated to the data to be processed based on the comparison results. Thus, based on whether a new data segment needs to be allocated to the data to be processed and the metadata corresponding to the data to be processed, the second metadata corresponding to the data to be processed on other clusters is updated first, and then the first metadata corresponding to the data to be processed on the target cluster is updated. This ensures that when the cluster type is other clusters, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is always in the latest state.

[0095] In one embodiment, an implementation method for processing data to be processed is provided, namely, "processing the data to be processed according to the updated metadata corresponding to the data to be processed" in S203 above, such as... Figure 4 As shown, it includes:

[0096] S401, determine the processing location and the type of processing request based on the updated metadata corresponding to the data to be processed. If the type of processing request indicates that the processing request is a write request, then proceed to step S402; if the type of processing request indicates that the processing request is a modification request, then proceed to step S403.

[0097] S402, write new data at the processing location.

[0098] S403, Modify the data to be processed.

[0099] In this embodiment, the target cluster can determine the processing location and the type of processing request based on the updated metadata corresponding to the data to be processed. The processing request type can be a write request or a modification request. If the processing request type indicates a write request, the processing location indicates the position where new data needs to be written, and thus, the target cluster can write new data at the processing location. For example, assuming the processing location is the end of the data to be processed, the target cluster can write new data to the end of the data to be processed. If the processing request type indicates a modification request, the processing location indicates the position of the data to be processed that needs to be modified, and thus, the target cluster can modify the data to be processed at that position.

[0100] The method described in this embodiment can accurately determine the processing location based on the updated metadata corresponding to the data to be processed. If the processing request is a write request, new data is written at the processing location; if the processing request is a modification request, the data to be processed is modified. In this way, the data to be processed can be processed adaptively according to different processing requests.

[0101] In the above embodiments, a method for reading target data is provided, namely... Figure 2 The data processing method of the embodiment, such as Figure 5 As shown, it also includes:

[0102] S204: When the target cluster receives a read request, it determines the storage location of the target data based on the metadata corresponding to the target data requested in the read request.

[0103] S205, Read the target data from the storage location of the target data.

[0104] The read request carries target data, which refers to the data in the cluster that the read request is requesting to be read. The metadata corresponding to the target data can include the location and capacity of the target data. The target data can be stored in any cluster of the distributed storage system. That is, if multiple clusters include a source server cluster and a remote server cluster, the target data can be stored in the source server cluster, or the target data can be stored in the remote server cluster, and both the source server cluster and the remote server cluster store the same metadata corresponding to the target data.

[0105] In this embodiment of the application, when the target cluster in the distributed storage system receives a read request, the target cluster can determine the metadata corresponding to the target data based on the target data requested in the read request. Therefore, the target cluster can determine the storage location of the target data based on the location of the target data included in the metadata. Afterwards, the target cluster can read the target data from its storage location.

[0106] For example, the target cluster can determine whether the target data is stored in the target cluster based on the metadata corresponding to the target data. If the metadata corresponding to the target data indicates that the target data is stored in the target cluster, the target cluster can determine the storage location of the target data in the target cluster based on the location of the target data included in the metadata. If the metadata corresponding to the target data indicates that the target data is stored in another cluster, the target cluster can send a second location retrieval request to the other cluster, instructing the other cluster, upon receiving the second location retrieval request, to determine the storage location of the target data on the other cluster based on the location of the target data included in the metadata, and send the storage location of the target data on the other cluster to the target cluster. Thus, the target cluster can receive the storage location of the target data on the other cluster returned by the other cluster.

[0107] In this embodiment, since the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is always up-to-date, when the target cluster receives a read request, it can accurately determine the storage location of the target data based on the accurate metadata corresponding to the target data requested by the read request. Thus, the target data can be accurately read from the storage location of the target data, thereby improving the data read success rate of each cluster in the distributed storage system.

[0108] In one embodiment, a distributed storage system is provided by combining the methods described in all the above embodiments, see [link to previous embodiment]. Figure 1 The distributed storage system shown includes a source server cluster and a remote server cluster, which communicate with each other.

[0109] The source server cluster is used to determine the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request when the source server cluster receives a processing request.

[0110] Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the source server cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0111] The data to be processed is processed according to the updated metadata corresponding to the data to be processed;

[0112] Alternatively, the remote server cluster is used to determine the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request when the remote server cluster receives a processing request.

[0113] Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the remote server cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0114] The data to be processed is processed based on the updated metadata corresponding to the data to be processed.

[0115] The steps described above have been explained in the foregoing embodiments and will not be repeated here.

[0116] In summary, based on all the above embodiments, this application also provides a complete data processing method, such as... Figure 6 As shown, the method includes:

[0117] S501: When the target cluster receives a processing request, it determines the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested in the processing request.

[0118] S502, determine the cluster type of the data to be processed based on the storage location, and compare the capacity of the data to be processed with the remaining capacity of the file where the storage location is located; if the cluster type is the target cluster, execute S503 or S504; if the cluster type is another cluster, execute S505 or S506.

[0119] S503, if the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then allocate a first data segment for the data to be processed on the target cluster, and update the first metadata corresponding to the data to be processed on the first data segment; and send a first update request message to other clusters; the first update request message is used to instruct other clusters to allocate a second data segment for the data to be processed, and update the second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0120] S504, if the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then update the first metadata on the target cluster according to the metadata corresponding to the data to be processed, and update the second metadata on other clusters according to the metadata corresponding to the data to be processed.

[0121] S505, if the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then allocate a third data segment for the data to be processed on other clusters, and update the second metadata corresponding to the data to be processed on the third data segment; and send a second update request message to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment for the data to be processed, and update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0122] S506, if the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then update the second metadata on other clusters according to the metadata corresponding to the data to be processed, and update the first metadata on the target cluster according to the metadata corresponding to the data to be processed.

[0123] S507, determine the processing location based on the updated metadata corresponding to the data to be processed;

[0124] S508: If the request to be processed is a write request, then new data is written at the processing location;

[0125] If the request is a modification request, then the data to be processed is modified.

[0126] S509: When the target cluster receives a read request, it determines the storage location of the target data based on the metadata corresponding to the target data requested in the read request.

[0127] S510 reads the target data from the storage location of the target data.

[0128] For example, assuming the distributed storage system includes a source server cluster (hereinafter referred to as the source cluster) and a remote server cluster (hereinafter referred to as the remote cluster), this embodiment of the application can support cross-cluster bilateral read / write operations on the shared data when the source cluster and the remote cluster share the same data and each stores the same metadata. Specifically, if the source cluster or the remote cluster receives a write request corresponding to a write operation, the local cluster receiving the write request can lock the file containing the data to be processed requested by the write request on its own side, and then notify the remote cluster to lock the file containing the data to be processed as well. Then, the local cluster can determine whether the data to be processed is stored on its own side based on the metadata corresponding to the data to be processed. If the data to be processed is stored on its own side, new data can be written directly on its own side; if the data to be processed is stored on the remote cluster, the local cluster can forward the message to the remote cluster to write new data in the remote cluster.

[0129] In one exemplary embodiment, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the source cluster write service in one embodiment, such as... Figure 8 As shown, Figure 8 This is a flowchart illustrating a data processing method when the target cluster is the source cluster, as shown in one embodiment. Here, oApp is the cluster's private client, oPara is the management module storing cluster metadata, and oStor is the management module storing cluster data. Combined with... Figure 7 and Figure 8 As shown, if the target cluster is the source cluster and the request is a write request, the data processing method flow is as follows: S601, when the source cluster oApp receives a write request corresponding to the file write service, the source cluster oApp adds a local lock to the file and notifies the remote cluster oApp across the cluster to add a local lock to the file. After the remote cluster oApp completes locking the file, the remote cluster oApp returns a first lock response message to the source cluster oApp. S602, the source cluster oApp obtains the metadata corresponding to the data to be processed requested by the write request from the source cluster oPara locally, and determines whether the data to be processed is in the local cluster (i.e., the source cluster) or the remote cluster based on the metadata.

[0130] If the data to be processed resides in the source cluster and a new data segment needs to be allocated, then in step S603, the source cluster oApp can request the allocation of a new data segment from its own cluster oPara. The source cluster oPara then allocates the new data segment for the data to be processed, updates the corresponding metadata, and writes it to disk. After the new data segment allocation is complete, it sends a first allocation response message to the source cluster oApp. Alternatively, the source cluster can also synchronize metadata information to the remote cluster via a background thread task. Specifically, in step S604, the source cluster oPara asynchronously synchronizes the new data segment allocation operation to the remote cluster oPara. The remote cluster oPara allocates a new data segment of the same size for the data to be processed, updates the corresponding metadata, and writes it to disk. After the new data segment allocation is complete, it sends a second allocation response message to the source cluster oPara. After both the source cluster and the source cluster's metadata have been updated, in step S605, the source cluster oApp writes new data to the local cluster oStor and uploads it to disk. After successfully writing the new data, the source cluster oStor sends the first write response message to the source cluster oApp and sends the write result to the user.

[0131] If the data to be processed is on a remote cluster and a new data segment needs to be allocated, then in step S606, the source cluster oApp can request the allocation of a new data segment from its local cluster oPara. The source cluster oPara then notifies the remote cluster oPara to allocate the new data segment synchronously. That is, the remote cluster oPara can allocate a new data segment for the data to be processed to update the corresponding metadata and remove it from disk. After the new data segment allocation is complete, it sends a third allocation response message to the source cluster oPara. In step S607, the source cluster oPara allocates a new data segment of the same size locally for the data to be processed to update the corresponding metadata and remove it from disk. After the new data segment allocation is complete, it sends a fourth allocation response message to the source cluster oApp. After both the source cluster and the source cluster's metadata have been updated, in step S608, the source cluster oApp notifies the remote cluster oApp to write new data across clusters. As a result, the remote cluster oApp writes the new data to the remote cluster oStor and pushes it to disk. After the remote cluster oStor finishes writing the new data, it sends a second write response message to the source cluster oApp and sends the write result to the user.

[0132] In another exemplary embodiment, such as Figure 9 As shown, Figure 9 This is a schematic diagram of a remote cluster write service in one embodiment, such as... Figure 10 As shown, Figure 10 This is a flowchart illustrating a data processing method when the target cluster is a remote cluster, as shown in one embodiment. (Combined with...) Figure 9 and Figure 10As shown, if the target cluster is a remote cluster and the request is a write request, the data processing method flow is as follows: S701, when the remote cluster oApp receives a write request corresponding to the file write service, it adds a local lock to the file and notifies the source cluster oApp across the cluster to add a local lock to the file. After the source cluster oApp completes locking the file, it returns a second lock response message to the remote cluster oApp. S702, the remote cluster oApp obtains the metadata corresponding to the data to be processed requested by the write request from the remote cluster oPara, and determines whether the data to be processed is in the local cluster (i.e., the remote cluster) or the source cluster based on the metadata.

[0133] If the data to be processed resides in the source cluster and a new data segment needs to be allocated, in step S703, the remote cluster oApp can notify the source cluster oApp across clusters that metadata needs to be updated. This allows the source cluster oApp to request the allocation of a new data segment from the source cluster oPara. The source cluster oPara then allocates the new data segment for the data to be processed, updates the corresponding metadata, and removes it from disk. After the new data segment allocation is complete, it sends a fifth allocation response message to the remote cluster oApp. Alternatively, the source cluster can synchronize metadata information to the remote cluster via a background thread task. Specifically, in step S704, the source cluster oPara asynchronously synchronizes the new data segment allocation operation to the remote cluster oPara. The remote cluster oPara allocates a new data segment of the same size for the data to be processed, updates the corresponding metadata, and removes it from disk. After the new data segment allocation is complete, it sends a sixth allocation response message to the source cluster oPara. After both the source cluster and the source cluster's metadata have been updated, in step S705, the remote cluster oApp notifies the source cluster oApp across clusters to write new data. As a result, the source cluster oApp writes the new data to the source cluster oStor and pushes it to disk. After the source cluster oStor finishes writing the new data, it sends a third write response message to the source cluster oApp. Then, the source cluster oApp sends a third write response message to the remote cluster oApp and sends the write result to the user.

[0134] If the data to be processed is in a remote cluster and a new data segment needs to be allocated, then in step S706, the remote cluster oApp can notify the source cluster oApp across clusters that it needs to update metadata, so that the source cluster oApp can request the allocation of a new data segment from the source cluster oPara. Then, the source cluster oPara notifies the remote cluster oPara to allocate the new data segment synchronously. That is, the remote cluster oPara can allocate a new data segment for the data to be processed to update the corresponding metadata and remove it from disk. After the new data segment allocation is complete, it sends a seventh allocation response message to the source cluster oPara. In step S707, the source cluster oPara allocates a new data segment of the same size locally for the data to be processed to update the corresponding metadata and remove it from disk. After the new data segment allocation is complete, it sends an eighth allocation response message to the source cluster oApp, which then sends the eighth allocation response message to the remote cluster oApp. After both the source cluster and the source cluster's metadata have been updated, in S708, the remote cluster oApp writes new data to the local cluster oStor and uploads it to disk. After successfully writing the new data, the remote cluster oStor sends a fourth write response message to the remote cluster oApp and sends the write result to the user.

[0135] In the above data processing method, the metadata corresponding to the data to be processed on the target cluster and other clusters can be synchronously updated. This ensures that the metadata corresponding to the data to be processed on each cluster in the distributed storage system remains consistent, thus avoiding difficulties in successfully reading data on both sides of the cluster due to data inconsistency. Therefore, the above method can improve the data read success rate of each cluster in the distributed storage system. This application embodiment targets multi-terminal clusters sharing the same data, allowing bilateral read / write operations on the same data on multiple terminal clusters, expanding the usage of cross-cluster services. Specifically, firstly, by simultaneously locking and protecting files in multiple terminal clusters, the consistency of data and metadata in the files can be guaranteed; secondly, when updating metadata through multi-terminal cluster clients, the update entry point must be uniformly the source cluster, ensuring the consistency of metadata in multiple terminal clusters; thirdly, by determining the storage location of the data to be processed through metadata information, new data can be efficiently and quickly written to the cluster to which the data to be processed belongs.

[0136] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0137] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0138] In one exemplary embodiment, such as Figure 11 As shown, a data processing apparatus is provided, including: a determining module 31, an updating module 32, and a processing module 33, wherein:

[0139] The determination module 31 is used to determine the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request when the target cluster receives a processing request.

[0140] The update module 32 is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located.

[0141] The processing module 33 is used to process the data to be processed based on the updated metadata corresponding to the data to be processed.

[0142] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed, and the update module 32 includes:

[0143] The comparison unit is used to determine the cluster type to which the data to be processed belongs based on the storage location, and to compare the capacity of the data to be processed with the remaining capacity of the file where the storage location is located;

[0144] The first update unit is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters according to the comparison result and the first update method when the cluster type is the target cluster.

[0145] The second update unit is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters according to the comparison results, when the cluster type is other clusters.

[0146] In one embodiment, the first updating unit is specifically used for:

[0147] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then a first data segment is allocated for the data to be processed on the target cluster, and the first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to other clusters; the first update request message is used to instruct other clusters to allocate a second data segment for the data to be processed, and to update the second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0148] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then update the first metadata on the target cluster according to the metadata corresponding to the data to be processed, and update the second metadata on other clusters according to the metadata corresponding to the data to be processed.

[0149] In one embodiment, the second updating unit is specifically used for:

[0150] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then a third data segment is allocated for the data to be processed on other clusters, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment for the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0151] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0152] In one embodiment, the processing module 33 includes:

[0153] The processing location determination unit is used to determine the processing location based on the updated metadata corresponding to the data to be processed.

[0154] The write unit is used to write new data at the processing location if the processing request is a write request;

[0155] The modification unit is used to modify the data to be processed if the processing request is a modification request.

[0156] In one embodiment, the data processing apparatus further includes:

[0157] The storage location determination module is used to determine the storage location of the target data based on the metadata corresponding to the target data requested by the read request when the target cluster receives a read request.

[0158] The read module is used to read the target data from its storage location.

[0159] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in the distributed storage system, or stored in software within the memory of the distributed storage system, so that the processor can invoke and execute the operations corresponding to each module.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0161] When the target cluster receives a processing request, it determines the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested in the processing request.

[0162] Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system.

[0163] The data to be processed is processed based on the updated metadata corresponding to the data to be processed.

[0164] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed. Based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system are updated. When the computer program is executed by the processor, it also implements the following steps:

[0165] Determine the cluster type to which the data to be processed belongs based on the storage location, and compare the capacity of the data to be processed with the remaining capacity of the file where the storage location is located;

[0166] If the cluster type is the target cluster, then based on the comparison results, the first update method is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters.

[0167] If the cluster type is other clusters, the second update method is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters, based on the comparison results.

[0168] In one embodiment, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters are updated using a first update method based on the comparison result. When the computer program is executed by the processor, the following steps are also implemented:

[0169] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then a first data segment is allocated for the data to be processed on the target cluster, and the first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to other clusters; the first update request message is used to instruct other clusters to allocate a second data segment for the data to be processed, and to update the second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0170] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then update the first metadata on the target cluster according to the metadata corresponding to the data to be processed, and update the second metadata on other clusters according to the metadata corresponding to the data to be processed.

[0171] In one embodiment, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters are updated using a second update method based on the comparison result. When the computer program is executed by the processor, the following steps are also implemented:

[0172] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, then a third data segment is allocated for the data to be processed on other clusters, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment for the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0173] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0174] In one embodiment, the data to be processed is processed according to the updated metadata corresponding to the data to be processed. When the computer program is executed by the processor, the following steps are also performed:

[0175] The processing location is determined based on the updated metadata corresponding to the data to be processed;

[0176] If the request is a write request, then write the new data at the processing location;

[0177] If the request is a modification request, then the data to be processed will be modified.

[0178] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0179] When the target cluster receives a read request, it determines the storage location of the target data based on the metadata corresponding to the target data requested in the read request.

[0180] Read the target data from the storage location of the target data.

[0181] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0184] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that: Applied to a distributed storage system, the distributed storage system includes multiple clusters, and the method includes: When the target cluster receives the processing request, the target cluster determines the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request; According to the attributes of the data to be processed and the remaining capacity of the file at the storage location, update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system; The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

2. The method according to claim 1, characterized in that The attribute of the data to be processed includes the capacity of the data to be processed, and updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attribute of the data to be processed and the remaining capacity of the file at the storage location, includes: Determining the type of cluster to which the data to be processed belongs according to the storage location, and comparing the capacity of the data to be processed with the remaining capacity of the file at the storage location; If the cluster type is the target cluster, then according to the comparison result, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other cluster are updated in a first updating manner; If the cluster type is the other cluster, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other cluster are updated in a second updating manner according to the comparison result.

3. The method according to claim 2, characterized in that The updating of the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters by using the first updating method according to the comparison result includes: If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other cluster; the first update request message is used to instruct the other cluster to allocate a second data segment to the data to be processed, and to update second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size; If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed.

4. The method according to claim 2, characterized in that: The updating, according to the comparison result, of the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters in a second updating manner includes: If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size; If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

5. The method according to any one of claims 1 to 4, characterized in that The processing of the data to be processed according to the updated metadata corresponding to the data to be processed includes: Determining a processing location according to the updated metadata corresponding to the data to be processed; If the processing request is a write request, writing new data at the processing location; If the processing request is a modification request, the data to be processed is modified.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When the target cluster receives a read request, determining a storage location of the target data according to metadata corresponding to the target data requested by the read request; The target data is read from a storage location of the target data.

7. A data processing device, characterized in that: The device comprises: A determination module, configured to determine, when the target cluster receives a processing request, a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request; An updating module, configured to update first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attributes of the data to be processed and the remaining capacity of the file at the storage location; The processing module is used to process the data to be processed according to the updated metadata corresponding to the data to be processed.

8. A distributed storage system, characterized in that: It includes a source server cluster and a remote server cluster, and the source server cluster and the remote server cluster are connected in communication; The source server cluster is used to determine the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request when the source server cluster receives the processing request; According to the attributes of the data to be processed and the remaining capacity of the file at the storage location, update the first metadata corresponding to the data to be processed on the source server cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system; Processing the data to be processed according to the updated metadata corresponding to the data to be processed; Alternatively, the remote server cluster is used to determine the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request when the remote server cluster receives the processing request; According to the attributes of the data to be processed and the remaining capacity of the file at the storage location, update the first metadata corresponding to the data to be processed on the remote server cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system; The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images