A data processing method and a data storage system

Abstract

The embodiment of the present application provides a kind of data processing method and data storage system, the data storage system is configured with the data storage partition for storing metadata, the method comprises: in the data storage system, create the protocol processing partition separated from the data storage partition;Wherein, the protocol processing partition is used to process access scheduling for metadata;According to the access condition for metadata, the current protocol processing partition is split to distribute access request to the multiple protocol processing partitions after splitting to service processing.Through the embodiment of the present application, the protocol processing layer of metadata is separated from the storage layer, and the access request is shunted by means of the light, fast splitting capacity of protocol processing partition, to cope with burst traffic and hotspot access and other high concurrent access scenarios, and the overall expansibility of service is improved.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a data processing method and a data storage system. Background Technology

[0002] Services such as PDS (Drive & Photo Service) contain massive amounts of metadata. To solve the problem of managing massive amounts of metadata, a partition management model can be used to create data storage partitions.

[0003] For services like PDS that experience high concurrency, system performance can be improved by splitting the data storage partitions. However, splitting data storage partitions requires a lot of resources and time, is difficult to perform at high frequencies, and has poor adaptability to high-concurrency access scenarios such as sudden traffic surges and hotspot access, thus affecting the overall scalability of the service. Summary of the Invention

[0004] In view of the above problems, a data processing method and data storage system are proposed to overcome or at least partially solve the above problems, including:

[0005] A data processing method, wherein a data storage system is configured with a data storage partition for storing metadata, the method comprising:

[0006] In the data storage system, a protocol processing partition is created that is separate from the data storage partition; the protocol processing partition is used to handle access scheduling for metadata.

[0007] Based on the access to metadata, the current protocol processing partition is split to distribute access requests to multiple split protocol processing partitions for service processing.

[0008] Optionally, it also includes:

[0009] During the current protocol's processing of access requests for metadata, the target metadata in the local cache is written to the log file, and the log file is synchronized to the data storage partition; the target metadata is the metadata that is updated based on the access request.

[0010] Optionally, before writing the target metadata from the local cache to the log file, the following steps are also included:

[0011] Read the required metadata from the data storage partition and store it in the local cache.

[0012] Optionally, before synchronizing the log files to the data storage partition, the following steps are also included:

[0013] Determine if the amount of metadata written to the log file exceeds a preset threshold, and if so, synchronize the log file to the data storage partition.

[0014] Optionally, the log files are synchronized to the data storage partition, including:

[0015] Based on the log file, update the index file of the data storage partition, and notify all data storage partitions to reload the updated index file.

[0016] Optionally, the current protocol processing partition may be split, including:

[0017] Split the log file of the partition being processed by the current protocol into multiple sub-log files;

[0018] One of the sub-log files is loaded into multiple new protocol processing partitions; these multiple new protocol processing partitions are used to provide services as the split protocol processing partitions.

[0019] Optionally, it also includes:

[0020] Record newly added target metadata; where the newly added target metadata refers to the target metadata added to the current protocol processing partition after the log file is split and before it is set to a disabled service state;

[0021] The newly added target metadata is synchronized to multiple new protocol processing partitions.

[0022] Optionally, after synchronizing the newly added target metadata to multiple new protocol processing partitions, the process also includes:

[0023] By using a two-phase commit approach, split completion messages are sent to multiple new protocol processing partitions, so that these new protocol processing partitions can begin providing services as the split protocol processing partitions.

[0024] Optionally, it also includes:

[0025] Set the current protocol processing partition to a disabled service state.

[0026] Optionally, it also includes:

[0027] Configure the number of replicas for the data storage partition based on the access patterns to metadata.

[0028] Optionally, the data storage partitions and protocol processing partitions in the data storage system adopt a distributed architecture, and the data storage partitions store metadata in key-value pairs.

[0029] A data storage system is provided, which is configured with a data storage partition for storing metadata and a computer program for execution within the data storage system, wherein the computer program, when executed by a processor, implements the data processing method described above.

[0030] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the data processing method described above.

[0031] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data processing method described above.

[0032] The embodiments of the present invention have the following advantages:

[0033] In this embodiment of the invention, a protocol processing partition separate from the data storage partition is created in the data storage system. The data storage partition is used to store metadata, and the protocol processing partition is used to handle access scheduling for metadata. Then, based on the access situation for metadata, the current protocol processing partition is split to distribute access requests to multiple split protocol processing partitions for service processing. This achieves the separation of the metadata protocol processing layer from the storage layer, and leverages the lightweight and fast splitting capability of the protocol processing partition to distribute access requests, thereby enabling the handling of high-concurrency access scenarios such as sudden traffic surges and hotspot access, and improving the overall scalability of the service. Attached Figure Description

[0034] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the steps of a data processing method provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of a system architecture provided in an embodiment of the present invention;

[0037] Figure 3 This is a flowchart of another data processing method provided in an embodiment of the present invention;

[0038] Figure 4 This is a flowchart of another data processing method provided in an embodiment of the present invention;

[0039] Figure 5This is a flowchart of another data processing method provided in an embodiment of the present invention;

[0040] Figure 6 This is a flowchart of another data processing method provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0042] In this embodiment of the invention, based on the storage and access characteristics of metadata in services such as PDS, an architecture that separates the protocol processing layer and the storage layer of metadata is adopted. By leveraging the lightweight and fast splitting capability of the protocol processing partition, access requests are distributed. This embodiment of the invention can manage massive amounts of metadata and effectively and promptly respond to high-concurrency access scenarios such as sudden and hot traffic, thus meeting the high availability, high performance, and high scalability requirements of services such as PDS.

[0043] For high availability, the rapid splitting of the protocol processing partition has a short impact time, and the multi-replica capability of the data storage partition ensures that if one replica fails, other replicas can still provide file access. For high performance, the splitting of the protocol processing partition results in more nodes providing services, leading to stronger performance; the multi-replica capability of the data processing partition works similarly. For high scalability, the splitting of the protocol processing partition and the multi-replica capability of the data processing partition enable linear scaling of system capacity.

[0044] Reference Figure 1 The diagram shows a flowchart of a data processing method according to an embodiment of the present invention. The data storage system can be a data storage system for metadata in services such as PDS.

[0045] To address the management challenges of massive amounts of metadata in services like PDS, a partitioned management model can be adopted in the data storage system. The data storage system can be configured with data storage partitions for storing metadata. In one example, the data storage partition can store metadata in key-value (KV) format, i.e., the data storage partition is a KV partition (KVPartition); the metadata can include related data based on a directory tree.

[0046] Specifically, it may include the following steps:

[0047] Step 101: In the data storage system, create a protocol processing partition that is separate from the data storage partition; wherein, the protocol processing partition is used to handle access scheduling for metadata.

[0048] For services like PDS that experience high concurrency, system performance can be improved by splitting the data storage partitions. However, splitting data storage partitions requires a lot of resources and time, is difficult to perform at high frequencies, and has poor adaptability to high-concurrency access scenarios such as sudden traffic surges and hotspot access, thus affecting the overall scalability of the service.

[0049] Specifically, in data storage partitions, the splitting of the original data storage partitions involves the splitting of metadata. After the splitting, the child data storage partitions must first load the data index of the metadata, and then obtain the latest metadata by replaying the log (RedoLog) of the parent data storage partition. This process consumes a lot of resources and time.

[0050] Based on this, the protocol processing layer of metadata can be separated from the storage layer. A partition management mode can be adopted for the protocol processing layer, creating a protocol processing partition that is separate from the data storage partition.

[0051] The protocol processing partition can be used to handle the access scheduling of metadata. The protocol processing partition is a logical partition and is the smallest scheduling unit for user IO access. It can be used to implement the access scheduling of metadata in services such as PDS. For example, the protocol processing partition is responsible for the process of creating files and then writing the newly created data to the data storage partition, which in turn manages the actual metadata.

[0052] Since the protocol only caches the most recently accessed metadata when processing partitions, splitting only requires splitting the most recently accessed metadata. This data volume is generally very small, often only in the MB range, and can be completed quickly.

[0053] In one embodiment of the present invention, to accommodate services such as PDS that have massive amounts of metadata, the data storage partitions and protocol processing partitions in the data storage system can adopt a distributed architecture. That is, multiple distributed data storage partitions and multiple distributed protocol processing partitions can be set up, such as... Figure 2 In the data storage system, configure multiple protocol processing partitions PP1 to PPn and multiple data storage partitions KP1 to KPn.

[0054] In one embodiment of the present invention, it further includes:

[0055] During the current protocol's processing of access requests for metadata, the target metadata in the local cache is written to the log file, and the log file is synchronized to the data storage partition; the target metadata is the metadata that is updated based on the access request.

[0056] After the protocol processing partition is created, all application access requests are handled by the protocol processing partition. The metadata in the data storage partition is read-only and will not be modified. All modifications are made on the protocol processing partition side.

[0057] During the current protocol processing partition's handling of access requests for metadata, the local cache of the current protocol processing partition can cache the most recently accessed metadata. The target metadata in the local cache can be determined. The target metadata is metadata that has been updated based on the access request, i.e., dirty data.

[0058] Specifically, metadata can consist of multiple attributes. If the value of any attribute is changed, then that metadata is considered dirty data, i.e., target metadata. For example, when an application accesses a file, it updates the last access time (atime) attribute. The value of the atime attribute changes because of the application's access, and the metadata of the accessed file is called dirty data, i.e., target metadata.

[0059] After determining the target metadata, it can be written to a log file. By synchronizing updated metadata from the protocol processing layer to the data storage layer, the protocol processing layer and the data storage layer are separated, allowing them to scale partitions independently. Furthermore, by storing the metadata of recently modified files in a unique log file for the current protocol processing partition, even if the current protocol processing partition crashes or fails over, the metadata can be recovered through these logs, thus providing services to the outside world.

[0060] In one example, if the target metadata cannot be written to the log file, the request processing is considered to have failed, the processing is terminated, and the result of the processing failure can be reported back to the application.

[0061] After writing the target metadata to the log file, the log file can be synchronized to the data storage partition. By synchronizing the log file carrying the updated metadata to the data storage partition, on the one hand, the latest metadata can be made visible to other protocol processing partitions through the data storage partition. On the other hand, after the log file is synchronized to the data storage partition, the current protocol processing partition no longer needs to maintain these log files, compressing the amount of log storage unique to the current protocol processing partition and ensuring that the amount of data split during the split is controllable.

[0062] In one embodiment of the present invention, before writing the target metadata in the local cache to the log file, the method further includes:

[0063] Read the required metadata from the data storage partition and store it in the local cache.

[0064] For an access request, the required metadata can be determined, and then it can be searched in the local cache of the current protocol processing partition first. If the required metadata is stored in the local cache, the metadata can be accessed directly. If the required metadata is not stored in the local cache, the required metadata can be read from the data storage partition, and then the required metadata can be updated in the local cache of the current protocol processing partition.

[0065] The protocol processing partition and the data storage partition can be deployed in the same cluster or in different clusters, but they may be loaded by different servers. They can communicate through RPC (Remote Procedure Call), and the protocol processing partition can read metadata from the data storage partition through RPC.

[0066] In one example, if the required metadata cannot be read from the data storage partition, the request processing is deemed to have failed, the processing is terminated, and the result of the processing failure can be reported back to the application.

[0067] In one embodiment of the present invention, before synchronizing the log file to the data storage partition, the method further includes:

[0068] Determine if the amount of metadata written to the log file exceeds a preset threshold, and if so, synchronize the log file to the data storage partition.

[0069] In the specific implementation, a preset quantity threshold can be set, such as 10MB. It can check whether the amount of metadata written to the log file is greater than the preset quantity threshold. If it is not greater than the preset quantity threshold, it can continue to write metadata. If it is greater than the preset quantity threshold, it can trigger a background operation to synchronize the log file to the data storage partition.

[0070] The following combination Figure 2 The following is an illustrative example of the partition processing read / write process described above:

[0071] a1. Check if the accessed metadata is cached locally. If it is, process the access request; otherwise, proceed to step a5.

[0072] a2. Write the dirty data (i.e. target metadata) during the request processing to the Log (i.e. log file). If the writing fails, the request processing fails and jumps to step a4.

[0073] a3. Check if the amount of dirty data in the Log exceeds a certain threshold (i.e., the preset threshold). If it does, trigger a background operation to synchronize the Log to the KV Partition.

[0074] a4. End the processing and return the result to the application;

[0075] a5. Read metadata from KV Partition via RPC. If successful, update the local cache and then jump to step a1. Otherwise, if the request processing fails, end the processing and return the result to the application.

[0076] In one embodiment of the present invention, synchronizing log files to a data storage partition includes:

[0077] Based on the log file, update the index file of the data storage partition, and notify all data storage partitions to reload the updated index file.

[0078] In practical applications, the existing index files in the data storage partition can be updated based on the log file carrying the target metadata, that is, merged into the existing index file to obtain all updated files. Then, all data storage partitions can be notified to reload the updated index files, thus completing the synchronization.

[0079] Specifically, log files can be serialized into storage files of a specified storage format. These storage files are used when the protocol processing partition synchronizes target metadata to the data storage partition. For example, log files can be serialized into Cpt (Checkpoint) files and then submitted to a specified service. This specified service is used to process the synchronization of log files from the protocol processing partition to the data storage partition, such as the Coordinator service.

[0080] For a specified service, it can merge a storage file of a specified storage format with an existing index file (MetaInfo, a file that stores a list of metadata index files in the storage data partition) to obtain a new index file. The specified service can then notify all data storage partitions to reload the new index folder and notify the current protocol processing partition that the log file synchronization is complete. After synchronization is complete, the current protocol processing partition can reclaim log space.

[0081] The following combination Figure 2 The process of synchronizing the log files described above is illustrated by example:

[0082] b1. Serialize the Log into a Cpt and then submit it to the Coordinator;

[0083] b2. The Coordinator merges the cpt and the current MetaInfo to obtain a new MetaInfo;

[0084] b3. The Coordinator notifies all KV Partitions to reload the new MetaInfo;

[0085] b4. The Coordinator notifies the Protocol Partition Log that synchronization is complete.

[0086] b5. When Protocol Partition receives the synchronization completion message, it reclaims the Log space, and the synchronization ends.

[0087] Since the synchronization operation of Cpt is idempotent, if any step from b1 to b4 fails, the current synchronization process can be terminated directly, and the process can be retried after a period of time.

[0088] Step 102: Based on the access to metadata, split the current protocol processing partition to distribute access requests to multiple split protocol processing partitions for service processing.

[0089] In practical applications, access to metadata in services such as PDS can be detected. Then, based on the access to metadata, the current protocol processing partition can be split. For example, in high-concurrency access scenarios such as sudden traffic surges and hotspot access, the current protocol processing partition can be quickly split, improving the system's ability to adapt to sudden traffic surges.

[0090] After splitting the current protocol processing partition, access traffic can be effectively distributed to more instances (i.e., the split protocol processing partition), improving the overall capacity of the system.

[0091] Since applications typically do not access many files in a short period of time, the amount of data synchronized when splitting the current protocol processing partition is usually small. Therefore, the split can be completed in a short time, thus ensuring that the protocol processing partition has the ability to cope with sudden traffic surges.

[0092] In one embodiment of the present invention, splitting the current protocol processing partition includes:

[0093] The log file of the current protocol processing partition is split into multiple sub-log files; one of the sub-log files is loaded into each of the multiple new protocol processing partitions; the multiple new protocol processing partitions are used to provide services as the split protocol processing partitions.

[0094] During the process of splitting the current protocol processing partition, the log file of the current protocol processing partition can be divided into multiple sub-log files. Then, multiple new protocol processing partitions can be selected as the split protocol processing partitions. One of the sub-log files is loaded into each new protocol processing partition, and then all the multiple sub-log files obtained by switching are migrated to multiple new protocol processing partitions.

[0095] Specifically, the current protocol processing partition can split the log file into multiple sub-log files, and then notify the specified service that it needs to split, such as the Coordinator. The specified service can then allocate multiple new protocol processing partitions to it and specify the initial log file of each new protocol processing partition from the multiple sub-log files.

[0096] After each new protocol processing partition has loaded its sub-log files, a specified service can be notified, which in turn can notify the current protocol processing partition which protocol processing partitions it has split into.

[0097] In one embodiment of the present invention, it further includes:

[0098] Record the newly added target metadata; synchronize the newly added target metadata to multiple new protocol processing partitions; wherein, the newly added target metadata is the target metadata added to the current protocol processing partition after the log file is split and before it is set to a disabled service state.

[0099] After the current protocol processing partition splits the log file, and before it is set to a disabled service state, that is, the current protocol processing partition is ready to split the log file to a new protocol processing partition, but it is still providing services, then the target metadata added during this period can be recorded, that is, the newly added dirty data.

[0100] After the new protocol processing partition loads the sub-log file, the current protocol processing partition learns which protocol processing partitions it has split into, and can then synchronize the newly added target metadata to the new protocol processing partitions. For example, if the current protocol processing partition P0 stores 100 newly added target metadata entries, it can use a two-equal-split method to synchronize 50 newly added target metadata entries to the new protocol processing partitions P1 and P2 respectively.

[0101] In one embodiment of the present invention, after synchronizing the newly added target metadata to multiple new protocol processing partitions, the method further includes:

[0102] By using a two-phase commit approach, split completion messages are sent to multiple new protocol processing partitions, so that these new protocol processing partitions can begin providing services as the split protocol processing partitions.

[0103] After the newly added target metadata is synchronized in the new protocol processing partition, a split completion message can be sent to multiple new protocol processing partitions through a two-phase commit, ensuring that each new protocol processing partition is aware that the split is complete. Once the new protocol processing partition is aware that the split is complete, it can then start providing services.

[0104] In one embodiment of the present invention, it further includes:

[0105] Set the current protocol processing partition to a disabled service state.

[0106] After the split is complete, the current protocol processing partition can be set to a disabled service state, and a specified service can be notified. The specified service can then reclaim the current protocol processing partition, and the split process ends.

[0107] In one example, during the splitting of the current protocol processing partition, a specified service can periodically check the progress of the split. If a failure or timeout occurs, the specified service can force a rollback of the split state, allowing the current protocol processing partition to continue providing services and discarding the new protocol processing partition.

[0108] The following combination Figure 2 The above splitting process is illustrated by example:

[0109] Assume that the Protocol Partition is split into two, with P0 being the partition to be split (i.e., the current protocol processing partition), and P1 and P2 being the sub-partitions after the split (i.e., the new protocol processing partitions), as follows:

[0110] c1 and P0 split Log (i.e., the log file) into Log1 and Log2 (i.e., sub-log files) according to the splitting rules, while recording the newly added dirty data (i.e., the newly added target metadata), and then notify the Coordinator that it needs to split.

[0111] c2, Coordinator allocates two new partitions P1 and P2, and then specifies the initial Logs of P1 and P2 as Log1 and Log2 respectively;

[0112] c3, P1, and P2 load Log1 and Log2 respectively, and notify the Coordinator when loading is complete;

[0113] c4. The Coordinator notifies P0 that the sub-partitions after the split are P1 and P2, respectively.

[0114] c5, P0 starts to synchronize the newly added dirty data to P1 and P2 respectively. After the synchronization is completed, a two-phase commit method is used to ensure that P1 and P2 can know that the split is completed. After receiving the message, P1 and P2 start the service.

[0115] c6 and P0 are set to a disabled service state to notify the Coordinator that the split is complete;

[0116] c7. The Coordinator recovers P0, and the splitting process is complete.

[0117] In one embodiment of the present invention, it further includes:

[0118] Configure the number of replicas for the data storage partition based on the access patterns to metadata.

[0119] In practical applications, since data storage partitions manage read-only metadata, when certain metadata becomes hot data, the number of replicas of the data storage partitions serving this hot data can be increased.

[0120] In this embodiment of the invention, a protocol processing partition separate from the data storage partition is created in the data storage system. The data storage partition is used to store metadata, and the protocol processing partition is used to handle access scheduling for metadata. Then, based on the access situation for metadata, the current protocol processing partition is split to distribute access requests to multiple split protocol processing partitions for service processing. This achieves the separation of the metadata protocol processing layer from the storage layer, and leverages the lightweight and fast splitting capability of the protocol processing partition to distribute access requests, thereby enabling the handling of high-concurrency access scenarios such as sudden traffic surges and hotspot access, and improving the overall scalability of the service.

[0121] Compared to managing directory tree metadata using a distributed shared block storage approach, it has the following advantages:

[0122] 1. In the distributed shared block storage approach, the protocol layer does not have log capability, resulting in longer IO paths and higher latency. However, in this embodiment of the invention, the protocol processing partition uses a temporary log mechanism to cache the most recently accessed data, which shortens the IO path, reduces latency, and improves write performance.

[0123] 2. In the distributed shared block storage approach, consistency depends on the capabilities of the shared blocks, which is generally complex to implement and has a long failover time. However, in this embodiment of the invention, when a failover reload occurs in the protocol processing partition, the recently accessed metadata can be loaded by replaying the log, thus reducing the impact of failover on access.

[0124] 3. In the distributed shared block storage approach, it is not possible to provide multiple replicas for hot data, and the ability to cope with hotspots and sudden traffic is limited. However, in this embodiment of the invention, by dynamically and flexibly configuring the number of replicas of data storage partitions, the query capability of metadata can be improved, thereby improving the read performance of metadata.

[0125] Reference Figure 3 The diagram illustrates a flowchart of another data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0126] Step 301: In the data storage system, create a protocol processing partition separate from the data storage partition; wherein the protocol processing partition is used to handle access scheduling for metadata.

[0127] Step 302: During the process of the current protocol processing partition processing access requests for metadata, the target metadata in the local cache is written to the log file, and the log file is synchronized to the data storage partition; wherein, the target metadata is metadata that is updated based on the access request.

[0128] After the protocol processing partition is created, all application access requests are handled by the protocol processing partition. The metadata in the data storage partition is read-only and will not be modified. All modifications are made on the protocol processing partition side.

[0129] During the current protocol processing partition's handling of access requests for metadata, the local cache of the current protocol processing partition can cache the most recently accessed metadata. The target metadata in the local cache can be determined. The target metadata is metadata that has been updated based on the access request, i.e., dirty data.

[0130] Specifically, metadata can consist of multiple attributes. If the value of any attribute is changed, then that metadata is considered dirty data, i.e., target metadata. For example, when an application accesses a file, it updates the last access time (atime) attribute. The value of the atime attribute changes because of the application's access, and the metadata of the accessed file is called dirty data, i.e., target metadata.

[0131] After determining the target metadata, it can be written to a log file. By synchronizing updated metadata from the protocol processing layer to the data storage layer, the protocol processing layer and the data storage layer are separated, allowing them to scale partitions independently. Furthermore, by storing the metadata of recently modified files in a unique log file for the current protocol processing partition, even if the current protocol processing partition crashes or fails over, the metadata can be recovered through these logs, thus providing services to the outside world.

[0132] In one example, if the target metadata cannot be written to the log file, the request processing is considered to have failed, the processing is terminated, and the result of the processing failure can be reported back to the application.

[0133] After writing the target metadata to the log file, the log file can be synchronized to the data storage partition. By synchronizing the log file carrying the updated metadata to the data storage partition, on the one hand, the latest metadata can be made visible to other protocol processing partitions through the data storage partition. On the other hand, after the log file is synchronized to the data storage partition, the current protocol processing partition no longer needs to maintain these log files, compressing the amount of log storage unique to the current protocol processing partition and ensuring that the amount of data split during the split is controllable.

[0134] Step 303: Based on the access to metadata, split the current protocol processing partition to distribute access requests to multiple split protocol processing partitions for service processing.

[0135] Reference Figure 4 The diagram illustrates a flowchart of another data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0136] Step 401: In the data storage system, create a protocol processing partition separate from the data storage partition; wherein the protocol processing partition is used to handle access scheduling for metadata.

[0137] Step 402: During the current protocol processing partition's handling of access requests for metadata, the required metadata is read from the data storage partition and stored in the local cache.

[0138] For an access request, the required metadata can be determined, and then it can be searched in the local cache of the current protocol processing partition first. If the required metadata is stored in the local cache, the metadata can be accessed directly. If the required metadata is not stored in the local cache, the required metadata can be read from the data storage partition, and then the required metadata can be updated in the local cache of the current protocol processing partition.

[0139] The protocol processing partition and the data storage partition can be deployed in the same cluster, but they may be loaded by different servers. They can communicate through RPC (Remote Procedure Call), and the protocol processing partition can read metadata from the data storage partition through RPC.

[0140] In one example, if the required metadata cannot be read from the data storage partition, the request processing is deemed to have failed, the processing is terminated, and the result of the processing failure can be reported back to the application.

[0141] Step 403: Write the target metadata in the local cache to the log file; wherein, the target metadata is the metadata that is updated based on the access request.

[0142] Step 404: Determine whether the amount of metadata written to the log file is greater than a preset threshold, and if the amount of metadata is greater than the preset threshold, synchronize the log file to the data storage partition.

[0143] In the specific implementation, a preset quantity threshold can be set, such as 10MB. It can check whether the amount of metadata written to the log file is greater than the preset quantity threshold. If it is not greater than the preset quantity threshold, it can continue to write metadata. If it is greater than the preset quantity threshold, it can trigger a background operation to synchronize the log file to the data storage partition.

[0144] The following combination Figure 2 The following is an illustrative example of the partition processing read / write process described above:

[0145] a1. Check if the accessed metadata is cached locally. If it is, process the access request; otherwise, proceed to step a5.

[0146] a2. Write the dirty data (i.e. target metadata) during the request processing to the Log (i.e. log file). If the writing fails, the request processing fails and jumps to step a4.

[0147] a3. Check if the amount of dirty data in the Log exceeds a certain threshold (i.e., the preset threshold). If it does, trigger a background operation to synchronize the Log to the KV Partition.

[0148] a4. End the processing and return the result to the application;

[0149] a5. Read metadata from KV Partition via RPC. If successful, update the local cache and then jump to step a1. Otherwise, if the request processing fails, end the processing and return the result to the application.

[0150] Step 405: Based on the access to metadata, split the current protocol processing partition to distribute access requests to multiple split protocol processing partitions for service processing.

[0151] Reference Figure 5 The diagram illustrates a flowchart of another data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0152] Step 501: In the data storage system, create a protocol processing partition separate from the data storage partition; wherein the protocol processing partition is used to handle access scheduling for metadata.

[0153] Step 502: During the current protocol processing partition's handling of access requests for metadata, the target metadata in the local cache is written to the log file; wherein, the target metadata is metadata that has been updated based on the access request.

[0154] Step 503: Update the index file of the data storage partition according to the log file, and notify all data storage partitions to reload the updated index file.

[0155] In practical applications, the existing index files in the data storage partition can be updated based on the log file carrying the target metadata, that is, merged into the existing index file to obtain all updated files. Then, all data storage partitions can be notified to reload the updated index files, thus completing the synchronization.

[0156] Specifically, log files can be serialized into storage files of a specified storage format. These storage files are used when the protocol processing partition synchronizes target metadata to the data storage partition. For example, log files can be serialized into Cpt (Checkpoint) files and then submitted to a specified service, such as the Coordinator, which is used to process the synchronization of log files from the protocol processing partition to the data storage partition.

[0157] For a specified service, it can merge a storage file of a specified storage format with the existing index file (MetaInfo, a file that stores a list of index files in the storage data partition) to obtain a new index file. The specified service can then notify all data storage partitions to reload the new index folder and notify the current protocol processing partition that the log file synchronization is complete. After synchronization is complete, the current protocol processing partition can reclaim log space.

[0158] The following combination Figure 2 The process of synchronizing the log files described above is illustrated by example:

[0159] b1. Serialize the Log into a Cpt and then submit it to the Coordinator;

[0160] b2. The Coordinator merges the cpt and the current MetaInfo to obtain a new MetaInfo;

[0161] b3. The Coordinator notifies all KV Partitions to reload the new MetaInfo;

[0162] b4. The Coordinator notifies the Protocol Partition Log that synchronization is complete.

[0163] b5. When Protocol Partition receives the synchronization completion message, it reclaims the Log space, and the synchronization ends.

[0164] Since the synchronization operation of Cpt is idempotent, if any step from b1 to b4 fails, the current synchronization process can be terminated directly, and the process can be retried after a period of time.

[0165] Step 504: Based on the access to metadata, split the current protocol processing partition to distribute access requests to multiple split protocol processing partitions for service processing.

[0166] Reference Figure 6 The diagram illustrates a flowchart of another data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0167] Step 601: In the data storage system, create a protocol processing partition separate from the data storage partition; wherein the protocol processing partition is used to handle access scheduling for metadata.

[0168] Step 602: During the process of the current protocol processing partition processing access requests for metadata, the target metadata in the local cache is written to the log file, and the log file is synchronized to the data storage partition; wherein, the target metadata is metadata that is updated based on the access request.

[0169] Step 603: Split the log file of the current protocol processing partition into multiple sub-log files, and load one of the sub-log files into each of the multiple new protocol processing partitions; wherein, the multiple new protocol processing partitions are used to provide services as the split protocol processing partitions.

[0170] During the process of splitting the current protocol processing partition, the log file of the current protocol processing partition can be divided into multiple sub-log files. Then, multiple new protocol processing partitions can be selected as the split protocol processing partitions. One of the sub-log files is loaded into each new protocol processing partition, and then all the multiple sub-log files obtained by switching are migrated to multiple new protocol processing partitions.

[0171] Specifically, the current protocol processing partition can split the log file into multiple sub-log files, and then notify the specified service that it needs to split, such as the Coordinator. The specified service can then allocate multiple new protocol processing partitions to it and specify the initial log file of each new protocol processing partition from the multiple sub-log files.

[0172] After each new protocol processing partition has loaded its sub-log files, a specified service can be notified, which in turn can notify the current protocol processing partition which protocol processing partitions it has split into.

[0173] Step 604: Record the newly added target metadata and synchronize the newly added target metadata to multiple new protocol processing partitions; wherein, the newly added target metadata is the target metadata added to the current protocol processing partition after the log file is split and before it is set to the disabled service state.

[0174] After the current protocol processing partition splits the log file, and before it is set to a disabled service state, that is, the current protocol processing partition is ready to split the log file to a new protocol processing partition, but it is still providing services, then the target metadata added during this period can be recorded, that is, the newly added dirty data.

[0175] After the new protocol processing partition loads the sub-log file, the current protocol processing partition learns which protocol processing partitions it has split into, and can then synchronize the newly added target metadata to the new protocol processing partitions. For example, if the current protocol processing partition P0 stores 100 newly added target metadata entries, it can use a two-equal-split method to synchronize 50 newly added target metadata entries to the new protocol processing partitions P1 and P2 respectively.

[0176] Step 605: Using a two-phase commit, send split completion messages to multiple new protocol processing partitions so that the multiple new protocol processing partitions can begin to provide services as the split protocol processing partitions.

[0177] After the newly added target metadata is synchronized in the new protocol processing partition, a split completion message can be sent to multiple new protocol processing partitions through a two-phase commit, ensuring that each new protocol processing partition is aware that the split is complete. Once the new protocol processing partition is aware that the split is complete, it can then start providing services.

[0178] Step 606: Set the current protocol processing partition to a disabled service state.

[0179] After the split is complete, the current protocol processing partition can be set to a disabled service state, and a specified service can be notified. The specified service can then reclaim the current protocol processing partition, and the split process ends.

[0180] In one example, during the splitting of the current protocol processing partition, a specified service can periodically check the progress of the split. If a failure or timeout occurs, the specified service can force a rollback of the split state, allowing the current protocol processing partition to continue providing services and discarding the new protocol processing partition.

[0181] The following combination Figure 2 The above splitting process is illustrated by example:

[0182] Assume that the Protocol Partition is split into two, with P0 being the partition to be split (i.e., the current protocol processing partition), and P1 and P2 being the sub-partitions after the split (i.e., the new protocol processing partitions), as follows:

[0183] c1 and P0 split Log (i.e., the log file) into Log1 and Log2 (i.e., sub-log files) according to the splitting rules, while recording the newly added dirty data (i.e., the newly added target metadata), and then notify the Coordinator that it needs to split.

[0184] c2, Coordinator allocates two new partitions P1 and P2, and then specifies the initial Logs of P1 and P2 as Log1 and Log2 respectively;

[0185] c3, P1, and P2 load Log1 and Log2 respectively, and notify the Coordinator when loading is complete;

[0186] c4. The Coordinator notifies P0 that the sub-partitions after the split are P1 and P2, respectively.

[0187] c5, P0 starts to synchronize the newly added dirty data to P1 and P2 respectively. After the synchronization is completed, a two-phase commit method is used to ensure that P1 and P2 can know that the split is completed. After receiving the message, P1 and P2 start the service.

[0188] c6 and P0 are set to a disabled service state to notify the Coordinator that the split is complete;

[0189] c7. The Coordinator recovers P0, and the splitting process is complete.

[0190] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0191] An embodiment of the present invention also provides a data storage system, which is configured with a data storage partition for storing metadata and a computer program for execution in the data storage system. When the computer program is executed by a processor, it implements the data processing method described above.

[0192] An embodiment of the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described data processing method.

[0193] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-described data processing method.

[0194] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0195] It should be noted that the user-related data involved in the embodiments of the present invention are all processed in accordance with the relevant data protection laws and policies of the country where the user is located, and with the authorization granted by the owner of the relevant device.

[0196] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0197] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0203] The above provides a detailed description of the data processing method and data storage system. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, The data storage system is configured with data storage partitions for storing metadata, and the method includes: In the data storage system, a protocol processing partition is created corresponding to the data storage partition; wherein the protocol processing partition is used to handle access scheduling for metadata; Based on the access patterns to metadata, the current protocol processing partition is split to distribute access requests to multiple split protocol processing partitions for service processing; wherein, the access patterns to metadata include high-concurrency access. The splitting of the current protocol processing partition includes: The log file of the currently processed partition is split into multiple sub-log files; One of the sub-log files is loaded into multiple new protocol processing partitions; wherein the multiple new protocol processing partitions are used to provide services as split protocol processing partitions.

2. The method according to claim 1, characterized in that, Also includes: During the process of the current protocol processing partition processing access requests for metadata, the target metadata in the local cache is written to the log file, and the log file is synchronized to the data storage partition; wherein, the target metadata is metadata that has been updated based on the access request.

3. The method according to claim 2, characterized in that, Before writing the target metadata from the local cache to the log file, the following is also included: The required metadata is read from the data storage partition and stored in the local cache.

4. The method according to claim 2, characterized in that, Before synchronizing the log file to the data storage partition, the method further includes: Determine whether the amount of metadata written to the log file is greater than a preset threshold, and if the amount of metadata is greater than the preset threshold, execute the step of synchronizing the log file to the data storage partition.

5. The method according to claim 2, characterized in that, The step of synchronizing the log file to the data storage partition includes: Based on the log file, the index file of the data storage partition is updated, and all data storage partitions are notified to reload the updated index file.

6. The method according to claim 1, characterized in that, Also includes: Record the newly added target metadata; wherein, the newly added target metadata is the target metadata added by the current protocol processing partition after the log file is split and before it is set to a service-disabled state; The newly added target metadata is synchronized to the multiple new protocol processing partitions.

7. The method according to claim 6, characterized in that, After synchronizing the newly added target metadata to the plurality of new protocol processing partitions, the method further includes: By using a two-phase commit method, a split completion message is sent to the plurality of new protocol processing partitions, so that the plurality of new protocol processing partitions can begin to provide services as the split protocol processing partitions.

8. The method according to claim 7, characterized in that, Also includes: Set the current protocol processing partition to a disabled service state.

9. The method according to claim 1, characterized in that, Also includes: Configure the number of replicas for the data storage partition based on the access patterns to metadata.

10. The method according to claim 1, characterized in that, The data storage system adopts a distributed architecture for data storage partitions and protocol processing partitions, and the data storage partitions store metadata in key-value pair format.

11. A data storage system, characterized in that, The data storage system is configured with a data storage partition for storing metadata and a computer program for execution in the data storage system, wherein the computer program, when executed by a processor, implements the data processing method as described in any one of claims 1 to 10.

12. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the data processing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the data processing method as described in any one of claims 1 to 10.

Images