Data processing method and apparatus, distributed storage system, storage medium and product

Abstract

The application relates to a data processing method and device, a distributed storage system, a storage medium and a product. The method comprises the following steps: when a target cluster receives a processing request, determining the storage location of to-be-processed data according to the metadata corresponding to the to-be-processed data requested by the processing request; updating the first metadata corresponding to the to-be-processed data on the target cluster and the second metadata corresponding to the to-be-processed data on other clusters in the distributed storage system according to the attribute of the to-be-processed data and the remaining capacity of the file where the storage location is located; and processing the to-be-processed data according to the updated metadata corresponding to the to-be-processed data. The method can ensure that the metadata corresponding to the to-be-processed data on each cluster in the distributed storage system is always consistent, thereby avoiding the difficulty in successfully reading data on the bilateral clusters due to the inconsistency of the data, so that the above method can improve the data reading success rate of each cluster in the distributed storage system.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data processing method, device, distributed storage system, storage medium and product. Background Art

[0002] A distributed storage system consists of multiple server clusters, such as a source server cluster and a remote server cluster. During operation, data in the source server cluster can be migrated or backed up to the remote server cluster to ensure data security within the distributed storage system.

[0003] However, after completing the above data migration or backup, the source server cluster or the remote server cluster has a low read success rate problem. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, device, distributed storage system, storage medium and product that can improve the reading success rate of a source server cluster or a remote server cluster in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, applied to a distributed storage system, wherein the distributed storage system includes multiple clusters, the method comprising:

[0006] When the target cluster receives the processing request, it determines the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request;

[0007] updating, according to the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system;

[0008] The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

[0009] In the above data processing method, the metadata corresponding to the data to be processed on the target cluster and other clusters can be updated synchronously. In this way, the metadata corresponding to the data to be processed on each cluster in the distributed storage system can be ensured to be always consistent, thereby avoiding the difficulty in successfully reading data on bilateral clusters due to data inconsistency. Therefore, the above method can improve the data reading success rate of each cluster in the distributed storage system.

[0010] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed, and updating, based on the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system includes:

[0011] Determining the cluster type to which the data to be processed belongs according to the storage location, and comparing the capacity of the data to be processed with the remaining capacity of the file at the storage location;

[0012] If the cluster type is the target cluster, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other cluster using the first updating method according to the comparison result;

[0013] If the cluster type is the other cluster, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other cluster are updated in a second updating manner according to the comparison result.

[0014] In this embodiment, the cluster type to which the data to be processed belongs can be determined based on the storage location, and the capacity of the data to be processed and the remaining capacity of the file at the storage location can be compared. Therefore, based on the comparison result and whether the cluster type is the target cluster or other cluster, the update method corresponding to the cluster type can be adopted to accurately synchronize the metadata corresponding to the data to be processed on the target cluster and other clusters. In this way, it can be ensured that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state.

[0015] In one embodiment, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters using the first updating method according to the comparison result includes:

[0016] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other cluster; the first update request message is used to instruct the other cluster to allocate a second data segment to the data to be processed, and to update second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0017] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed.

[0018] In this embodiment, it is possible to accurately determine whether it is necessary to allocate new data segments for the data to be processed based on the comparison results. Thus, based on whether it is necessary to allocate new data segments for the data to be processed and the metadata corresponding to the data to be processed, the first metadata corresponding to the data to be processed on the target cluster can be updated first, and then the second metadata corresponding to the data to be processed on other clusters can be updated. This ensures that when the cluster type is the target cluster, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state.

[0019] In one embodiment, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters in a second updating manner according to the comparison result includes:

[0020] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0021] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0022] In this embodiment, it is possible to accurately determine whether it is necessary to allocate new data segments for the data to be processed based on the comparison results. Thus, based on whether it is necessary to allocate new data segments for the data to be processed and the metadata corresponding to the data to be processed, the second metadata corresponding to the data to be processed on other clusters can be updated first, and then the first metadata corresponding to the data to be processed on the target cluster can be updated. This ensures that when the cluster type is other clusters, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and in the latest state.

[0023] In one embodiment, processing the data to be processed according to the updated metadata corresponding to the data to be processed includes:

[0024] determining a processing location according to the updated metadata corresponding to the data to be processed;

[0025] If the processing request is a write request, writing new data at the processing location;

[0026] If the processing request is a modification request, the data to be processed is modified.

[0027] In this embodiment, the processing location can be accurately determined based on the updated metadata corresponding to the data to be processed. If the processing request is a write request, the new data is written to the processing location; if the processing request is a modify request, the data to be processed is modified. In this way, the data to be processed can be adaptively processed according to different processing requests.

[0028] In one embodiment, the method further comprises:

[0029] When the target cluster receives a read request, determining a storage location of the target data according to metadata corresponding to the target data requested by the read request;

[0030] The target data is read from a storage location of the target data.

[0031] In this embodiment, since the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state, when the target cluster receives a read request, it can accurately determine the storage location of the target data based on the accurate metadata corresponding to the target data requested by the read request, thereby accurately reading the target data from the storage location of the target data, thereby improving the data reading success rate of each cluster in the distributed storage system.

[0032] In a second aspect, the present application further provides a data processing device, comprising:

[0033] a determination module, configured to, when a target cluster receives a processing request, determine a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request;

[0034] An updating module, configured to update, according to the attributes of the data to be processed and the remaining capacity of the file at the storage location, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system;

[0035] The processing module is used to process the data to be processed according to the updated metadata corresponding to the data to be processed.

[0036] In a third aspect, the present application further provides a distributed storage system. The distributed storage system includes a source server cluster and a remote server cluster, wherein the source server cluster and the remote server cluster are in communication connection with each other;

[0037] The source server cluster is configured to determine, when the source server cluster receives a processing request, a storage location of the data to be processed based on metadata corresponding to the data to be processed requested by the processing request;

[0038] updating, based on the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the source server cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system;

[0039] Processing the data to be processed according to the updated metadata corresponding to the data to be processed;

[0040] Alternatively, the remote server cluster is configured to determine, when the remote server cluster receives a processing request, a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request;

[0041] updating, according to the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the remote server cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system;

[0042] The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect.

[0044] In a fifth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect.

[0045] The above-mentioned data processing method, device, distributed storage system, storage medium and product, when the target cluster receives a processing request, determines the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request; updates the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attributes of the data to be processed and the remaining capacity of the file where the storage location is located; and processes the data to be processed according to the updated metadata corresponding to the data to be processed. The embodiment of the present application can synchronously update the metadata corresponding to the data to be processed on the target cluster and other clusters, so that it can ensure that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always consistent, thereby avoiding the difficulty in successfully reading data on the bilateral cluster due to data inconsistency. Therefore, the above-mentioned method can improve the data reading success rate of each cluster in the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A diagram of an application environment of a data processing method in one embodiment;

[0048] Figure 2 1 is a flow chart of a data processing method in one embodiment;

[0049] Figure 3 Schematic diagram of a process of metadata updating step in one embodiment;

[0050] Figure 4 Schematic diagram of a flow chart of the steps for processing data to be processed in one embodiment;

[0051] Figure 5 is a flow chart of a data processing method in another embodiment;

[0052] Figure 6 is a flow chart of a data processing method in an optional embodiment;

[0053] Figure 7 This is a schematic diagram of a source-side cluster write service in one embodiment;

[0054] Figure 8 1 is a flow chart of a data processing method when the target cluster is a source cluster in one embodiment;

[0055] Figure 9 A schematic diagram of a remote cluster write service in one embodiment;

[0056] Figure 10 1 is a flow chart of a data processing method when the target cluster is a remote cluster in one embodiment;

[0057] Figure 11 FIG. 4 is a structural block diagram of a data processing device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0060] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0061] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] A distributed storage system consists of multiple server clusters, such as a source server cluster and a remote server cluster. During operation, data in the source server cluster can be migrated or backed up across clusters to a remote server cluster, ensuring the security of the data stored in the distributed storage system.

[0063] When storing data across clusters, traditional technology writes a copy of the data in both the source server cluster and the remote server cluster, thereby achieving synchronous backup of data between the source server cluster and the remote server cluster. The above backup method requires two copies of data storage space in the distributed storage system. However, since the data in a file cannot be modified on two clusters in the distributed storage system at the same time, if a modification or write operation is performed on the data in either the source server cluster or the target server cluster, the data in the source server cluster and the target server cluster will be different, resulting in the inability to successfully read the same data when reading the data in the source server cluster or the remote server cluster. Therefore, after the traditional technology completes the above data migration or backup, the source server cluster or the remote server cluster has the problem of low read success rate.

[0064] After introducing the background technology of the data processing method provided by the embodiment of the present application, the following briefly describes the implementation environment involved in the data processing method provided by the embodiment of the present application. The data processing method provided by the embodiment of the present application can be applied to Figure 1 The distributed storage system shown in FIG. The distributed storage system includes multiple types of server clusters, and different types of server clusters can communicate with each other, perform data transmission and information exchange operations, such as, Figure 1 The distributed storage system shown includes a source server cluster 101 and a remote server cluster 102. Bilateral business processing can be implemented between the source server cluster 101 and the remote server cluster 102. For example, both the source server cluster and the remote server cluster support read and write services. Various types of server clusters can be server clusters composed of multiple physical servers. Each physical server can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, and the like. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like.

[0065] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the distributed storage system to which the solution of the present application is applied. The specific distributed storage system may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0066] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 The application in the distributed storage system is taken as an example to illustrate, including the following steps:

[0067] S201 : When a target cluster receives a processing request, it determines a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request.

[0068] Among them, the target cluster refers to any cluster in the distributed storage system, and the processing request may include but is not limited to data processing requests such as write requests and modification requests. The processing request carries data to be processed, and the data to be processed refers to the data in the cluster requested to be processed by the processing request. For example, if the processing request is a write request, the data to be processed requested by the processing request refers to the data that needs to be written to the corresponding location in the cluster. The metadata corresponding to the data to be processed may include the location of the data to be processed and the capacity of the data to be processed. The storage location of the data to be processed can be in any cluster in the distributed storage system, that is, it can be understood that if multiple clusters include a source server cluster and a remote server cluster, the data to be processed can be stored in the source server cluster, or the data to be processed can also be stored in the remote server cluster, and the source server cluster and the remote server cluster both store the same metadata corresponding to the data to be processed.

[0069] In an embodiment of the present application, when a target cluster in a distributed storage system receives a processing request, the target cluster can determine the metadata corresponding to the to-be-processed data based on the to-be-processed data requested by the processing request, thereby determining the storage location of the to-be-processed data based on the location of the to-be-processed data included in the metadata corresponding to the to-be-processed data. Exemplarily, the target cluster can determine whether the to-be-processed data is stored in the target cluster based on the metadata corresponding to the to-be-processed data. If the metadata corresponding to the to-be-processed data indicates that the to-be-processed data is stored in the target cluster, the target cluster can determine the storage location of the to-be-processed data in the target cluster based on the location of the to-be-processed data included in the metadata. If the metadata corresponding to the to-be-processed data indicates that the to-be-processed data is stored in other clusters, the target cluster can send a location acquisition request to the other cluster to instruct the other cluster to determine the storage location of the to-be-processed data on the other cluster based on the location of the to-be-processed data included in the metadata after receiving the location acquisition request, and send the storage location of the to-be-processed data on the other cluster to the target cluster, thereby allowing the target cluster to receive the storage location of the to-be-processed data on the other cluster returned by the other cluster. Wherein, the other cluster refers to the cluster other than the target cluster in the distributed storage system.

[0070] S202 , updating first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to attributes of the data to be processed and remaining capacity of the file at the storage location.

[0071] Among them, the attributes of the data to be processed may include but are not limited to the capacity of the data to be processed, the amount of resources occupied by the data to be processed, etc., and the remaining capacity of the file where the storage location is located refers to the size of the remaining storage space corresponding to the file where the storage location of the data to be processed is located. The first metadata refers to the metadata corresponding to the data to be processed on the target cluster, and the second metadata refers to the metadata corresponding to the data to be processed on other clusters. It should be noted that the embodiment of the present application can synchronously update the metadata corresponding to the data to be processed on each cluster in the distributed storage system, so that it can ensure that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same.

[0072] In an embodiment of the present application, the target cluster can determine the attributes of the data to be processed and the remaining capacity of the file at the storage location based on the metadata corresponding to the data to be processed, and then determine whether it is necessary to allocate a new data segment for the data to be processed based on the attributes of the data to be processed and the remaining capacity of the file at the storage location. Thus, the target cluster can update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system based on the storage location of the data to be processed and whether it is necessary to allocate a new data segment for the data to be processed. Optionally, the target cluster can first update the first metadata corresponding to the data to be processed on the target cluster, and then update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system; or, the target cluster can first update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, and then update the first metadata corresponding to the data to be processed on the target cluster. Of course, the embodiment of the present application does not limit the order of updating the first metadata and the second metadata.

[0073] S203: Process the data to be processed according to the updated metadata corresponding to the data to be processed.

[0074] The embodiments of the present application involve two methods of processing data to be processed. The first method is that if the metadata corresponding to the data to be processed indicates that the data to be processed is stored in the target cluster, the target cluster can directly process the data to be processed at the storage location according to the updated metadata corresponding to the data to be processed; the second method is that if the metadata corresponding to the data to be processed indicates that the data to be processed is stored in other clusters, the target cluster can send a processing request message to the other cluster to instruct the other cluster to process the data to be processed at the storage location according to the updated metadata corresponding to the data to be processed after receiving the processing request message, and send a processing response message to the target cluster, so that the target cluster can receive the processing response message returned by the other cluster, wherein the processing response message is used to indicate that the processing of the data to be processed has been completed. Of course, the embodiments of the present application do not limit the specific implementation method of processing the data to be processed.

[0075] In the above data processing method, when the target cluster receives a processing request, the storage location of the data to be processed is determined based on the metadata corresponding to the data to be processed requested by the processing request; based on the attributes of the data to be processed and the remaining capacity of the file where the storage location is located, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system are updated; and the data to be processed is processed based on the updated metadata corresponding to the data to be processed. The embodiment of the present application can synchronously update the metadata corresponding to the data to be processed on the target cluster and other clusters, so that the metadata corresponding to the data to be processed on each cluster in the distributed storage system can be ensured to be consistent, thereby avoiding the difficulty in successfully reading data on the bilateral cluster due to data inconsistency. Therefore, the above method can improve the data reading success rate of each cluster in the distributed storage system.

[0076] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed. Based on this, an implementation method for updating the first metadata and the second metadata is provided, namely, "updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attributes of the data to be processed and the remaining capacity of the file at the storage location" in the above S202. Figure 3 Shown, including:

[0077] S301 : determining the cluster type to which the data to be processed belongs according to the storage location, and comparing the capacity of the data to be processed with the remaining capacity of the file at the storage location.

[0078] The attributes of the data to be processed include the capacity of the data to be processed, where the capacity of the data to be processed refers to the size of the storage space occupied by the data to be processed. The cluster type may include a target cluster or other clusters.

[0079] In an embodiment of the present application, the target cluster can determine whether the cluster to which the data to be processed belongs is the target cluster or another cluster based on the storage location of the data to be processed, and the target cluster can compare the capacity of the data to be processed with the remaining capacity of the file at the storage location to obtain a comparison result. The comparison result is used to indicate that the capacity of the data to be processed is greater than the remaining capacity of the file, or to indicate that the capacity of the data to be processed is not greater than the remaining capacity of the file. Of course, in an embodiment of the present application, there is no limitation on the order of determining the cluster type and obtaining the comparison result.

[0080] S302: If the cluster type is a target cluster, then according to the comparison result, a first updating method is used to update first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters.

[0081] In an embodiment of the present application, if it is determined that the cluster type to which the data to be processed belongs is a target cluster, the target cluster can determine whether it is necessary to allocate a new data segment for the data to be processed based on the comparison result. Thus, based on whether it is necessary to allocate a new data segment for the data to be processed, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters can be updated using the first update method. Among them, the first update method is: first update the first metadata corresponding to the data to be processed on the target cluster, and then update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system. That is, it can be understood that if the cluster type is a local cluster, that is, the cluster that receives the processing request is the same as the cluster type to which the data to be processed belongs, then the local cluster can first update the first metadata corresponding to the data to be processed on the local cluster, and then update the second metadata corresponding to the data to be processed on the cluster other than the local cluster (that is, the opposite cluster).

[0082] S303: If the cluster type is other clusters, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters are updated using the second updating method according to the comparison result.

[0083] In an embodiment of the present application, if it is determined that the cluster type to which the data to be processed belongs is another cluster, the target cluster can determine whether it is necessary to allocate a new data segment for the data to be processed based on the comparison result. Thus, based on whether it is necessary to allocate a new data segment for the data to be processed, the second update method can be used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters. Among them, the second update method is: first update the second metadata corresponding to the data to be processed on other clusters in the distributed storage system, and then update the first metadata corresponding to the data to be processed on the target cluster. That is, it can be understood that if the cluster type is a peer cluster, that is, the cluster that receives the processing request is different from the cluster type to which the data to be processed belongs, then the local cluster can first update the second metadata corresponding to the data to be processed on the peer cluster, and then update the first metadata corresponding to the data to be processed on the cluster other than the peer cluster (that is, the local cluster).

[0084] In this embodiment, the cluster type to which the data to be processed belongs can be determined based on the storage location, and the capacity of the data to be processed and the remaining capacity of the file at the storage location can be compared. Therefore, based on the comparison result and whether the cluster type is the target cluster or other cluster, the update method corresponding to the cluster type can be adopted to accurately synchronize the metadata corresponding to the data to be processed on the target cluster and other clusters. In this way, it can be ensured that the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state.

[0085] In one embodiment, an implementation method for updating the first metadata and the second metadata using the first update method is provided, namely, "updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters using the first update method according to the comparison result" in S302, including:

[0086] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and the first metadata corresponding to the data to be processed is updated on the first data segment. A first update request message is also sent to the other clusters. The first update request message is used to instruct the other clusters to allocate a second data segment to the data to be processed and update the second metadata corresponding to the data to be processed on the second data segment. The first data segment and the second data segment have the same size. If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the first metadata on the target cluster is updated based on the metadata corresponding to the data to be processed, and the second metadata on the other clusters is updated based on the metadata corresponding to the data to be processed.

[0087] In an embodiment of the present application, the target cluster can compare the capacity of the data to be processed with the remaining capacity of the file at the storage location to obtain a comparison result. If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, it indicates that a new data segment needs to be allocated for the data to be processed. In this case, the target cluster can determine the size of the data segment to be allocated based on the difference between the capacity of the data to be processed and the remaining capacity of the file, allocate a first data segment to the data to be processed on the target cluster, and update the first metadata corresponding to the data to be processed on the first data segment. After the first metadata update is completed, the target cluster can send a first update request message to other clusters so that the other clusters, after receiving the first update request message, allocate a second data segment to the data to be processed based on the size of the first data segment and update the second metadata corresponding to the data to be processed on the second data segment. The size of the data segment to be allocated, the size of the first data segment, and the size of the second data segment are all the same. The first update request message is used to instruct the other clusters to allocate the second data segment to the data to be processed and update the second metadata corresponding to the data to be processed on the second data segment. A data segment refers to a slice of data in a file, and each sliced ​​data storage unit can be regarded as a data segment.

[0088] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, it means that there is no need to allocate a new data segment for the data to be processed. At this time, the target cluster can determine the processing position and processing capacity corresponding to the processing request based on the metadata corresponding to the data to be processed, and thus update the first metadata on the target cluster based on the processing position and processing capacity. After the first metadata update is completed, a third update request message is sent to other clusters so that the other clusters update the second metadata on the other clusters according to the metadata corresponding to the data to be processed after receiving the third update request message. The third update request message is used to instruct other clusters to update the second metadata on the other clusters according to the metadata corresponding to the data to be processed. It should be noted that the updated first metadata and the updated second metadata are the same.

[0089] In this embodiment, it is possible to accurately determine whether it is necessary to allocate new data segments for the data to be processed based on the comparison results. Thus, based on whether it is necessary to allocate new data segments for the data to be processed and the metadata corresponding to the data to be processed, the first metadata corresponding to the data to be processed on the target cluster can be updated first, and then the second metadata corresponding to the data to be processed on other clusters can be updated. This ensures that when the cluster type is the target cluster, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state.

[0090] In one embodiment, an implementation method for updating the first metadata and the second metadata using the second update method is provided, namely, "updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters using the second update method according to the comparison result" in S303 above, including:

[0091] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment. A second update request message is sent to the target cluster. The second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed and to update the first metadata corresponding to the data to be processed on the fourth data segment. The third and fourth data segments have the same size. If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the second metadata on the other cluster is updated based on the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated based on the metadata corresponding to the data to be processed.

[0092] In an embodiment of the present application, the target cluster can compare the capacity of the data to be processed and the remaining capacity of the file where the storage location is located to obtain a comparison result. If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, it means that a new data segment needs to be allocated for the data to be processed. At this time, the target cluster can determine the size of the data segment to be allocated based on the difference between the capacity of the data to be processed and the remaining capacity of the file, and send a fourth update request message to other clusters, so that after receiving the fourth update request message, the other clusters allocate a third data segment for the data to be processed on the other clusters according to the size of the data segment to be allocated, and update the second metadata corresponding to the data to be processed on the third data segment. After the second metadata update is completed, the other clusters can send a second update request message to the target cluster, so that after receiving the second update request message, the target cluster allocates a fourth data segment for the data to be processed according to the size of the third data segment, and updates the first metadata corresponding to the data to be processed on the fourth data segment. If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, it means that there is no need to allocate a new data segment for the data to be processed. At this time, the target cluster can send a fifth update request message to other clusters, so that other clusters can update the second metadata on other clusters according to the metadata corresponding to the data to be processed after receiving the fifth update request message. After the second metadata update is completed, the target cluster can determine the processing position and processing capacity corresponding to the processing request based on the metadata corresponding to the data to be processed, thereby updating the first metadata on the target cluster according to the processing position and processing capacity. Among them, the fifth update request message is used to instruct other clusters to update the second metadata on other clusters according to the metadata corresponding to the data to be processed. It should be noted that the updated first metadata and the updated second metadata are the same.

[0093] The size of the data segment to be allocated, the size of the third data segment, and the size of the fourth data segment are the same. The second update request message is used to instruct the target cluster to allocate the fourth data segment for the data to be processed and to update the first metadata corresponding to the data to be processed in the fourth data segment. The fourth update request message is used to instruct the other cluster to allocate the third data segment for the data to be processed and to update the second metadata corresponding to the data to be processed in the third data segment.

[0094] In this embodiment, it is possible to accurately determine whether it is necessary to allocate new data segments for the data to be processed based on the comparison results. Thus, based on whether it is necessary to allocate new data segments for the data to be processed and the metadata corresponding to the data to be processed, the second metadata corresponding to the data to be processed on other clusters can be updated first, and then the first metadata corresponding to the data to be processed on the target cluster can be updated. This ensures that when the cluster type is other clusters, the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and in the latest state.

[0095] In one embodiment, a method for implementing processing of the data to be processed is provided, namely, "processing the data to be processed according to the updated metadata corresponding to the data to be processed" in the above S203, such as Figure 4 Shown, including:

[0096] S401, determine the processing location and the type of processing request based on the updated metadata corresponding to the data to be processed. If the type of processing request indicates that the processing request is a write request, execute step S402; if the type of processing request indicates that the processing request is a modify request, execute step S403.

[0097] S402, writing new data at the processing location.

[0098] S403: Modify the data to be processed.

[0099] In an embodiment of the present application, the target cluster can determine the processing location based on the updated metadata corresponding to the data to be processed, and can determine the type of processing request. Among them, the type of processing request can be a write request type or a modification request type. If the type of processing request indicates that the processing request is a write request, the processing location indicates the location where the new data needs to be written, so that the target cluster can write the new data at the processing location. For example, assuming that the processing location is the end of the data to be processed, then the target cluster can write the new data to the end of the data to be processed. If the type of processing request indicates that the processing request is a modification request, the processing location indicates the location of the data to be processed that needs to be modified, so that the target cluster can modify the data to be processed at the location where the data to be processed is located.

[0100] The method described in this embodiment accurately determines the processing location based on the updated metadata corresponding to the data to be processed. If the processing request is a write request, the new data is written to the processing location; if the processing request is a modify request, the data to be processed is modified. In this way, the data to be processed can be adaptively processed according to different processing requests.

[0101] In the above embodiment, a method for reading target data is provided, namely Figure 2 The data processing method of the embodiment, such as Figure 5 As shown, it also includes:

[0102] S204 : When the target cluster receives the read request, it determines the storage location of the target data according to metadata corresponding to the target data requested by the read request.

[0103] S205: Read the target data from the storage location of the target data.

[0104] The read request carries target data, which refers to the data in the cluster that the read request requests. The metadata corresponding to the target data may include the location and capacity of the target data. The target data can be stored in any cluster in the distributed storage system. In other words, if multiple clusters include a source server cluster and a remote server cluster, the target data can be stored in the source server cluster, or the target data can be stored in the remote server cluster, and both the source server cluster and the remote server cluster store the same metadata corresponding to the target data.

[0105] In an embodiment of the present application, when a target cluster in a distributed storage system receives a read request, the target cluster can determine metadata corresponding to the target data based on the target data requested by the read request. Thus, the target cluster can determine the storage location of the target data based on the location of the target data included in the metadata corresponding to the target data. The target cluster can then read the target data from the storage location of the target data.

[0106] Exemplarily, the target cluster can determine whether the target data is stored in the target cluster based on the metadata corresponding to the target data. If the metadata corresponding to the target data indicates that the target data is stored in the target cluster, the target cluster can determine the storage location of the target data in the target cluster based on the location of the target data included in the above metadata. If the metadata corresponding to the target data indicates that the target data is stored in other clusters, the target cluster can send a second location acquisition request to the other cluster to instruct the other cluster to determine the storage location of the target data on the other cluster based on the location of the target data included in the above metadata after receiving the second location acquisition request, and send the storage location of the target data on the other cluster to the target cluster, so that the target cluster can receive the storage location of the target data on the other cluster returned by the other cluster.

[0107] In this embodiment, since the metadata corresponding to the data to be processed on each cluster in the distributed storage system is always the same and is in the latest state, when the target cluster receives a read request, it can accurately determine the storage location of the target data based on the accurate metadata corresponding to the target data requested by the read request, thereby accurately reading the target data from the storage location of the target data, thereby improving the data reading success rate of each cluster in the distributed storage system.

[0108] In one embodiment, a distributed storage system is provided by combining the methods described in all the above embodiments. Figure 1 The distributed storage system shown includes a source server cluster and a remote server cluster, wherein the source server cluster and the remote server cluster are communicatively connected;

[0109] The source server cluster is configured to determine, when the source server cluster receives a processing request, a storage location of the data to be processed based on metadata corresponding to the data to be processed requested by the processing request;

[0110] updating, based on the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the source server cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system;

[0111] Processing the data to be processed according to the updated metadata corresponding to the data to be processed;

[0112] Alternatively, the remote server cluster is configured to determine, when the remote server cluster receives a processing request, a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request;

[0113] updating, according to the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the remote server cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system;

[0114] The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

[0115] The above steps have been described in the previous embodiments and will not be repeated here.

[0116] In summary of all the above embodiments, this application also provides a complete data processing method, such as Figure 6 As shown, the method includes:

[0117] S501, when the target cluster receives a processing request, determines the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request;

[0118] S502, determining the cluster type to which the data to be processed belongs based on the storage location, and comparing the capacity of the data to be processed with the remaining capacity of the file at the storage location; if the cluster type is the target cluster, executing S503 or S504; if the cluster type is other clusters, executing S505 or S506;

[0119] S503: If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other cluster; the first update request message is used to instruct the other cluster to allocate a second data segment to the data to be processed, and to update second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0120] S504: If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then updating the first metadata on the target cluster according to the metadata corresponding to the data to be processed, and updating the second metadata on the other clusters according to the metadata corresponding to the data to be processed;

[0121] S505: If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0122] S506, if the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, then updating the second metadata on the other clusters according to the metadata corresponding to the data to be processed, and updating the first metadata on the target cluster according to the metadata corresponding to the data to be processed;

[0123] S507, determining a processing location according to the updated metadata corresponding to the data to be processed;

[0124] S508, if the processing request is a write request, write new data at the processing location;

[0125] If the processing request is a modification request, the data to be processed will be modified;

[0126] S509, when the target cluster receives the read request, it determines the storage location of the target data according to the metadata corresponding to the target data requested by the read request;

[0127] S510: Read target data from a storage location of the target data.

[0128] For example, assuming that the distributed storage system includes a source-end server cluster (referred to as the source cluster) and a remote server cluster (referred to as the remote cluster), then, the embodiment of the present application can support cross-cluster bilateral read / write services for a shared copy of data when the source cluster and the remote cluster share a copy of data, and the source cluster and the remote cluster each store a copy of the same metadata. Specifically, if the source cluster or the remote cluster receives a write request corresponding to the write service, the local cluster that receives the write request can lock the file where the data to be processed requested by the write request is located in the local cluster, and then notify the remote cluster to lock the file where the data to be processed is located. Then, the local cluster can determine whether the data to be processed is stored in the local cluster based on the metadata corresponding to the data to be processed. If the data to be processed is stored in the local cluster, the new data can be written directly to the local cluster; if the data to be processed is stored in the remote cluster, the local cluster can forward the message to the remote cluster to write the new data in the remote cluster.

[0129] In an exemplary embodiment, Figure 7 As shown, Figure 7 This is a schematic diagram of a source-side cluster write service in one embodiment, as shown in FIG. Figure 8 As shown, Figure 8 The following is a flow chart of a data processing method when the target cluster is a source cluster in an embodiment. In which, oApp is the private client of the cluster, oPara is the management module for storing cluster metadata, and oStor is the management module for storing cluster data. Figure 7 and Figure 8 As shown, if the target cluster is the source cluster and the processing request is a write request, the data processing method proceeds as follows: S601: When the source cluster oApp receives a write request corresponding to a file write service, the source cluster oApp applies a local lock to the file and notifies the remote cluster oApp service across clusters to apply a local lock to the file. After the remote cluster oApp completes locking the file, the remote cluster oApp returns a first lock response message to the source cluster oApp. S602: The source cluster oApp obtains metadata corresponding to the to-be-processed data requested by the write request from the source cluster oPara locally and determines whether the to-be-processed data is in the local cluster (i.e., the source cluster) or the remote cluster based on the metadata.

[0130] If the data to be processed is in the source cluster and a new data segment needs to be allocated, at this time, S603, the source cluster oApp can apply to the cluster oPara for allocation of a new data segment, so that the source cluster oPara allocates a new data segment for the data to be processed to update the metadata corresponding to the data to be processed and download it to disk, and after the allocation of the new data segment is completed, it sends a first allocation response message to the source cluster oApp. In addition, the source cluster can also synchronize metadata information to the remote cluster through background thread tasks. Specifically, S604, the source cluster oPara synchronizes the operation of allocating new data segments to the remote cluster oPara in an asynchronous manner. The remote cluster oPara allocates a new data segment of the same size for the data to be processed to update the metadata corresponding to the data to be processed and download it to disk, and after the allocation of the new data segment is completed, it sends a second allocation response message to the source cluster oPara. After the source cluster and the metadata of the source cluster are updated, S605, the source cluster oApp writes the new data to the current cluster oStor and saves it to disk. After successfully writing the new data, the source cluster oStor sends a first write response message to the source cluster oApp and sends the write result to the user.

[0131] If the data to be processed is in the remote cluster and a new data segment needs to be allocated, at this time, S606, the source cluster oApp can apply to the local cluster oPara for allocation of the new data segment. Thus, the source cluster oPara notifies the remote cluster oPara to allocate the new data segment in a synchronous manner. That is, the remote cluster oPara can allocate the new data segment for the data to be processed to update the metadata corresponding to the data to be processed and save it to disk. After the allocation of the new data segment is completed, it sends a third allocation response message to the source cluster oPara. S607, the source cluster oPara locally allocates a new data segment of the same size for the data to be processed to update the metadata corresponding to the data to be processed and save it to disk. After the allocation of the new data segment is completed, it sends a fourth allocation response message to the source cluster oApp. After the source cluster and the metadata of the source cluster are updated, S608, the source cluster oApp notifies the remote cluster oApp across clusters to write new data, so that the remote cluster oApp writes the new data to the remote cluster oStor and saves it to the disk. After the remote cluster oStor completes writing the new data, it sends a second write response message to the source cluster oApp and sends the write result to the user.

[0132] In another exemplary embodiment, Figure 9 As shown, Figure 9 FIG. 1 is a schematic diagram of a remote cluster write service in one embodiment, such as Figure 10 As shown, Figure 10 FIG. 1 is a flow chart of a data processing method when the target cluster is a remote cluster in an embodiment. Figure 9 and Figure 10As shown, if the target cluster is a remote cluster and the processing request is a write request, the data processing method proceeds as follows: S701: When the remote cluster oApp receives a write request corresponding to a file write service, the remote cluster oApp applies a local lock to the file and notifies the source cluster oApp service across clusters to apply a local lock to the file. After the source cluster oApp completes locking the file, the source cluster oApp returns a second lock response message to the remote cluster oApp. S702: The remote cluster oApp obtains metadata corresponding to the pending data requested by the write request from the remote cluster oPara locally and determines, based on this metadata, whether the pending data is in the local cluster (i.e., the remote cluster) or the source cluster.

[0133] If the data to be processed is in the source cluster and a new data segment needs to be allocated, at this time, S703, the remote cluster oApp can notify the source cluster oApp across clusters that the metadata needs to be updated, so that the source cluster oApp applies to the source cluster oPara for allocation of new data segments, so that the source cluster oPara allocates new data segments for the data to be processed to update the metadata corresponding to the data to be processed and download it to disk, and after the allocation of the new data segment is completed, it sends a fifth allocation response message to the remote cluster oApp. In addition, the source cluster can also synchronize metadata information to the remote cluster through background thread tasks. Specifically, S704, the source cluster oPara synchronizes the operation of allocating new data segments to the remote cluster oPara in an asynchronous manner. The remote cluster oPara allocates new data segments of the same size for the data to be processed to update the metadata corresponding to the data to be processed and download it to disk, and after the allocation of the new data segment is completed, it sends a sixth allocation response message to the source cluster oPara. After the metadata of the source cluster and the source cluster are updated, S705, the remote cluster oApp notifies the source cluster oApp across clusters to write new data, so that the source cluster oApp writes the new data to the source cluster oStor and saves it to the disk. After the source cluster oStor completes writing the new data, it sends a third write response message to the source cluster oApp, and then the source cluster oApp sends a third write response message to the remote cluster oApp, and sends the write result to the user.

[0134] If the data to be processed is in the remote cluster and a new data segment needs to be allocated, at this time, S706, the remote cluster oApp can notify the source cluster oApp across clusters that the metadata needs to be updated, so that the source cluster oApp applies to the source cluster oPara for allocation of new data segments. Thus, the source cluster oPara notifies the remote cluster oPara to allocate new data segments in a synchronous manner, that is, the remote cluster oPara can allocate new data segments for the data to be processed to update the metadata corresponding to the data to be processed and download it to disk, and after the allocation of the new data segments is completed, it sends the seventh allocation response message to the source cluster oPara. S707, the source cluster oPara locally allocates new data segments of the same size for the data to be processed to update the metadata corresponding to the data to be processed and download it to disk, and after the allocation of the new data segments is completed, it sends the eighth allocation response message to the source cluster oApp, and the source cluster oApp sends the eighth allocation response message to the remote cluster oApp. After the source cluster and the metadata of the source cluster are updated, S708, the remote cluster oApp writes the new data to the local cluster oStor and saves it to the disk. After successfully writing the new data, the remote cluster oStor sends a fourth write response message to the remote cluster oApp and sends the write result to the user.

[0135] In the above data processing method, the metadata corresponding to the data to be processed on the target cluster and other clusters can be updated synchronously. In this way, the metadata corresponding to the data to be processed on each cluster in the distributed storage system can be guaranteed to be consistent, thereby avoiding the difficulty in successfully reading data on the bilateral cluster due to data inconsistency. Therefore, the above method can improve the data reading success rate of each cluster in the distributed storage system. The embodiment of the present application is aimed at multi-terminal clusters that share a copy of data. Bilateral read / write business operations can be performed on the same copy of data in the multi-terminal clusters respectively, expanding the use of cross-cluster business. Specifically, first, by locking and protecting files in the multi-terminal clusters at the same time, the consistency of data and metadata in the file can be guaranteed; second, when updating metadata through the multi-terminal cluster client, it is necessary to uniformly use the source cluster as the update entry to ensure the consistency of metadata in the multi-terminal cluster; third, the storage location of the data to be processed is determined by the metadata information, and the new data can be written to the cluster to which the data to be processed belongs efficiently and quickly.

[0136] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0137] Based on the same inventive concept, the present application also provides a data processing device for implementing the aforementioned data processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0138] In an exemplary embodiment, Figure 11 As shown, a data processing device is provided, including: a determination module 31, an update module 32 and a processing module 33, wherein:

[0139] The determination module 31 is configured to determine, when the target cluster receives a processing request, a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request.

[0140] The updating module 32 is configured to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attributes of the data to be processed and the remaining capacity of the file at the storage location.

[0141] The processing module 33 is configured to process the data to be processed according to the updated metadata corresponding to the data to be processed.

[0142] In one embodiment, the attribute of the data to be processed includes the capacity of the data to be processed, and the updating module 32 includes:

[0143] a comparing unit, configured to determine the type of cluster to which the data to be processed belongs based on the storage location, and to compare the capacity of the data to be processed with the remaining capacity of the file at the storage location;

[0144] A first updating unit is configured to update, when the cluster type is a target cluster, first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters using a first updating method according to the comparison result;

[0145] The second updating unit is configured to update, when the cluster type is other clusters, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters by adopting a second updating method according to the comparison result.

[0146] In one embodiment, the first updating unit is specifically configured to:

[0147] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other cluster; the first update request message is used to instruct the other cluster to allocate a second data segment to the data to be processed, and update second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0148] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed.

[0149] In one embodiment, the second updating unit is specifically configured to:

[0150] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0151] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0152] In one embodiment, the processing module 33 includes:

[0153] a processing position determining unit, configured to determine a processing position according to updated metadata corresponding to the data to be processed;

[0154] a writing unit, configured to write new data at the processing location if the processing request is a write request;

[0155] The modification unit is configured to modify the data to be processed if the processing request is a modification request.

[0156] In one embodiment, the data processing device further includes:

[0157] A storage location determination module is used to determine the storage location of the target data according to the metadata corresponding to the target data requested by the read request when the target cluster receives the read request;

[0158] The reading module is used to read the target data from the storage location of the target data.

[0159] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the distributed storage system in hardware form, or can be stored in the memory of the distributed storage system in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0161] When the target cluster receives the processing request, it determines the storage location of the data to be processed based on the metadata corresponding to the data to be processed requested by the processing request;

[0162] Update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system according to the attributes of the data to be processed and the remaining capacity of the file at the storage location;

[0163] The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

[0164] In one embodiment, the attributes of the data to be processed include the capacity of the data to be processed. Based on the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the target cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system are updated. When the computer program is executed by the processor, the following steps are further implemented:

[0165] Determine the cluster type to which the data to be processed belongs based on the storage location, and compare the capacity of the data to be processed with the remaining capacity of the file at the storage location;

[0166] If the cluster type is a target cluster, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters are updated using the first update method according to the comparison result;

[0167] If the cluster type is other clusters, the second updating method is used to update the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters according to the comparison result.

[0168] In one embodiment, based on the comparison result, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters are updated using the first updating method. When the computer program is executed by the processor, the following steps are further implemented:

[0169] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other cluster; the first update request message is used to instruct the other cluster to allocate a second data segment to the data to be processed, and update second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size;

[0170] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed.

[0171] In one embodiment, based on the comparison result, the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters are updated using the second updating method. When the computer program is executed by the processor, the following steps are further implemented:

[0172] If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size;

[0173] If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the second metadata on other clusters is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

[0174] In one embodiment, the data to be processed is processed according to the updated metadata corresponding to the data to be processed, and when the computer program is executed by the processor, the following steps are further implemented:

[0175] Determine a processing location based on updated metadata corresponding to the data to be processed;

[0176] If the processing request is a write request, new data is written at the processing location;

[0177] If the processing request is a modification request, the data to be processed is modified.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0179] When the target cluster receives a read request, it determines the storage location of the target data based on the metadata corresponding to the target data requested by the read request;

[0180] Read the target data from the storage location of the target data.

[0181] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0182] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0183] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0184] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: Applied to a distributed storage system including multiple clusters, the method includes: When the target cluster receives the processing request, it determines the storage location of the data to be processed according to the metadata corresponding to the data to be processed requested by the processing request; Determining the cluster type to which the data to be processed belongs according to the storage location, and comparing the capacity of the data to be processed with the remaining capacity of the file at the storage location; If the cluster type is the target cluster, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on other clusters in the distributed storage system using the first updating method according to the comparison result; If the cluster type is the other cluster, updating the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other cluster using a second updating method according to the comparison result; the first updating method and the second updating method are different; The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

2. The method according to claim 1, characterized in that The updating of the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters by using the first updating method according to the comparison result includes: If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a first data segment is allocated to the data to be processed on the target cluster, and first metadata corresponding to the data to be processed is updated on the first data segment; and a first update request message is sent to the other cluster; the first update request message is used to instruct the other cluster to allocate a second data segment to the data to be processed, and to update second metadata corresponding to the data to be processed on the second data segment; the first data segment and the second data segment have the same size; If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed, and the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed.

3. The method according to claim 1, characterized in that The updating of the first metadata corresponding to the data to be processed on the target cluster and the second metadata corresponding to the data to be processed on the other clusters by using the second updating method according to the comparison result includes: If the comparison result indicates that the capacity of the data to be processed is greater than the remaining capacity of the file, a third data segment is allocated to the data to be processed on the other cluster, and the second metadata corresponding to the data to be processed is updated on the third data segment; and a second update request message is sent to the target cluster; the second update request message is used to instruct the target cluster to allocate a fourth data segment to the data to be processed, and to update the first metadata corresponding to the data to be processed on the fourth data segment; the third data segment and the fourth data segment have the same size; If the comparison result indicates that the capacity of the data to be processed is not greater than the remaining capacity of the file, the second metadata on the other cluster is updated according to the metadata corresponding to the data to be processed, and the first metadata on the target cluster is updated according to the metadata corresponding to the data to be processed.

4. The method according to any one of claims 1 to 3, characterized in that The processing of the data to be processed according to the updated metadata corresponding to the data to be processed includes: determining a processing location according to the updated metadata corresponding to the data to be processed; If the processing request is a write request, writing new data at the processing location; If the processing request is a modification request, the data to be processed is modified.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When the target cluster receives a read request, determining a storage location of the target data according to metadata corresponding to the target data requested by the read request; The target data is read from a storage location of the target data.

6. A data processing device, characterized in that: Applied to a distributed storage system including multiple clusters, the device includes: a determination module, configured to, when a target cluster receives a processing request, determine a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request; a comparing unit, configured to determine the type of cluster to which the data to be processed belongs according to the storage location, and to compare the capacity of the data to be processed with the remaining capacity of the file at the storage location; a first updating unit, configured to update, when the cluster type is the target cluster, first metadata corresponding to the to-be-processed data on the target cluster and second metadata corresponding to the to-be-processed data on other clusters in the distributed storage system using a first updating method according to a comparison result; a second updating unit, configured to update, when the cluster type is the other cluster, the first metadata corresponding to the to-be-processed data on the target cluster and the second metadata corresponding to the to-be-processed data on the other cluster using a second updating method according to the comparison result; the first updating method and the second updating method being different; The processing module is used to process the data to be processed according to the updated metadata corresponding to the data to be processed.

7. A distributed storage system, characterized in that: It includes a source server cluster and a remote server cluster, wherein the source server cluster and the remote server cluster are connected in communication; The source server cluster is configured to determine, when the source server cluster receives a processing request, a storage location of the data to be processed based on metadata corresponding to the data to be processed requested by the processing request; updating, based on the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the source server cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system; Processing the data to be processed according to the updated metadata corresponding to the data to be processed; Alternatively, the remote server cluster is configured to determine, when the remote server cluster receives a processing request, a storage location of the data to be processed according to metadata corresponding to the data to be processed requested by the processing request; updating, according to the attributes of the data to be processed and the remaining capacity of the file at the storage location, first metadata corresponding to the data to be processed on the remote server cluster and second metadata corresponding to the data to be processed on other clusters in the distributed storage system; The data to be processed is processed according to the updated metadata corresponding to the data to be processed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images