Data processing method of key value storage system and key value storage system

Abstract

The invention relates to a data processing method of a key value storage system and the key value storage system. The key value storage system comprises a computing node, a distributed log service and a configuration center, the distributed log service is used for storing data in each log file, and the configuration center is used for storing configuration data and metadata of each log file; the computing node receives a data processing command; processing at least one of a hash table data structure stored in a memory, data stored in each log file, cache data stored in a cache layer of the memory, configuration data and metadata; the hash table data structure comprises multiple pairs of keys and values, and the values are log information for storing corresponding data. The method serves as a storage system based on the distributed log service, does not need to highly depend on a memory, and greatly improves the upper limit value of data capacity. Based on a designed hash table data structure, high availability of data is ensured. Separation of calculation and storage is realized, the utilization rate of resources is improved, and the flexibility and efficiency of calculation are improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method for a key-value storage system and a key-value storage system. Background Art

[0002] Currently, the key-value storage system Redis is a key-value storage solution that is highly dependent on memory. All of its operating data is stored in memory, which has obvious capacity limitations. In addition, its computing resources and storage resources are bound together, and resource utilization is inflexible, which limits the flexibility and efficiency of computing. Summary of the invention

[0003] Based on this, it is necessary to provide a data processing method and a key-value storage system for the above technical problems, so as to realize the separation of computing and storage and improve the utilization of resources.

[0004] In a first aspect, the present application provides a data processing method of a key-value storage system, wherein the key-value storage system includes a computing node, a distributed log service, and a configuration center, wherein the distributed log service is used to store data in each log file, and the configuration center is used to store configuration data and metadata of each log file; the method is applied to the computing node, and includes:

[0005] receiving data processing commands;

[0006] Based on the data processing command, at least one of the hash table data structure stored in the memory, the data stored in each log file, the cache data stored in the cache layer of the memory, the configuration data and the metadata is processed; wherein the hash table data structure includes multiple pairs of keys and values, and the values ​​are log information storing corresponding data.

[0007] In one embodiment, the data processing command includes an access data command; the method further includes:

[0008] Based on the key corresponding to the access data command, determining corresponding first log information in the hash table data structure;

[0009] Based on the first log information, searching for corresponding data in the cache layer;

[0010] If no data is found, the distributed log service interface is called to pass in the first log information;

[0011] Receiving target data fed back by the distributed log service based on the first log information;

[0012] According to the first log information and the target data, the cache data is stored in the cache layer, and the target data is fed back to the client that sends the data access command.

[0013] In one embodiment, the data processing command includes a write data command; the method further includes:

[0014] Based on the write data command, the interface of the distributed log service is called, and the data to be written corresponding to the write data command is passed in, so as to instruct the distributed log service to store the data to be written in an idle log file, and feedback the second log information storing the data to be written;

[0015] Receive the second log information fed back by the distributed log service;

[0016] According to the key corresponding to the data to be written and the second log information, a key-value pair is added to the hash table data structure, and according to the second log information and the data to be written, the cache data is stored in the cache layer.

[0017] In one embodiment, the data processing command includes a delete command; the method further includes:

[0018] Based on the key corresponding to the delete command, determining corresponding third log information in the hash table data structure;

[0019] Calling the interface of the distributed log service, passing in the third log information, to instruct the distributed log service to determine the data to be deleted based on the third log information, and writing information corresponding to the data to be deleted in the tombstone identification log;

[0020] According to the key corresponding to the delete command, the key-value pair is deleted in the hash table data structure, and according to the third log information, the cache data stored in the cache layer is deleted.

[0021] In one embodiment, the method further comprises:

[0022] Get the metadata of the kth log file; the kth log file is the log file currently written by the distributed log service;

[0023] If the metadata meets the preset switching conditions, the distributed log service is called to create a new k+2th log file, and the k+1th log file is used as the currently written log file;

[0024] Synchronize the metadata of the kth log file to the configuration center.

[0025] In one embodiment, the method further comprises:

[0026] Count the invalid data information and valid data information of each log file;

[0027] Determine multiple log files that meet preset merging conditions according to the invalid data information and the valid data information;

[0028] Call the distributed log service to merge the data stored in multiple log files.

[0029] In one embodiment, calling a distributed log service to merge data stored in multiple log files includes:

[0030] Determine, according to the hash table data structure, fourth log information corresponding to valid data in the plurality of log files;

[0031] Calling the interface of the distributed log service, creating a new first log file, and writing the valid data corresponding to the fourth log information into the first log file;

[0032] The metadata corresponding to the first log file is synchronized to the configuration center, and the hash table data structure is updated according to the first log file.

[0033] In one embodiment, the method further comprises:

[0034] At startup, obtain checkpoint information from the configuration center;

[0035] Call the distributed log service interface to read the log data in the log file corresponding to the checkpoint information;

[0036] According to the log data, a hash table data structure is constructed;

[0037] After calling the distributed log service and merging the data stored in the multiple log files, the method further includes:

[0038] Call the distributed log service, create a new second log file, and write log index information into the second log file;

[0039] Generate target checkpoint information based on the metadata of the second log file and synchronize it to the configuration center;

[0040] Delete the historical checkpoint information in the configuration center;

[0041] Call the distributed log service to delete the log files associated with the historical checkpoint information from multiple log files.

[0042] In one embodiment, receiving a data processing command includes:

[0043] Receive Redis commands sent by the client;

[0044] Use access control lists to verify client permissions;

[0045] After the permission verification is passed, the Redis command is converted into a data processing command.

[0046] In the second aspect, the present application also provides a key-value storage system, which includes computing nodes, a distributed log service and a configuration center, the distributed log service is used to store data in each log file, and the configuration center is used to store configuration data and metadata of each log file; when the computing node executes a computer program, the steps of the method described in the first aspect above are implemented.

[0047] The data processing method and key-value storage system of the above-mentioned key-value storage system, the key-value storage system includes a computing node, a distributed log service and a configuration center, the distributed log service is used to store data in each log file, and the configuration center is used to store configuration data and metadata of each log file; the computing node receives a data processing command; based on the data processing command, at least one of the hash table data structure stored in the memory, the data stored in each log file, the cache data stored in the cache layer of the memory, the configuration data and the metadata is processed; wherein, the hash table data structure includes multiple pairs of keys and values, and the value is the log information for storing the corresponding data. Through the above-mentioned method, based on the distributed log service as a storage system, there is no need to rely heavily on the memory, which greatly improves the upper limit of the data capacity. Based on the designed hash table data structure, the high availability of data can be guaranteed. Based on the computing node and the distributed log service, the separation of computing and storage is realized, the utilization rate of resources is improved, and the flexibility and efficiency of computing are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] FIG1 is an application environment diagram of a data processing method of a key-value storage system in one embodiment;

[0050] FIG2 is a schematic diagram of a flow chart of a data processing method of a key-value storage system in one embodiment;

[0051] FIG3 is a schematic diagram of a module of a computing node in one embodiment;

[0052] FIG4 is a schematic diagram of a process of accessing data in one embodiment;

[0053] FIG5 is a schematic diagram of a search using a HASH TABLE in one embodiment;

[0054] FIG6 is a schematic flow chart of a data processing method of a key-value storage system in another embodiment;

[0055] FIG7 is a schematic diagram of a process of merging logs in one embodiment;

[0056] FIG8 is a schematic diagram of a checkpoint execution flow in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, the first log information may be referred to as the second log information, and similarly, the second log information may be referred to as the first log information. Both the first log information and the second log information are log information, but they may not be the same log information.

[0059] The data processing method of the key-value storage system provided in the embodiment of the present application can be applied to the application environment as shown in Figure 1. Among them, the key-value storage system includes a computing node 10, a distributed log service 20 and a configuration center 30, the distributed log service 20 is used to store data in each log file, and the configuration center 30 is used to store configuration data and metadata of each log file. The client communicates with the computing node 10 through the network, sends a data processing command to the computing node 10, and the computing node 10 executes the steps of the data processing method of the key-value storage system provided in the present application, and feeds back the processing results to the client. Optionally, the computing node 10 can synchronously receive data processing commands sent by multiple clients, and multi-threaded concurrent processing of different data processing requirements. Optionally, the distributed log service 20 includes multiple log nodes, each of which manages one or more log files.

[0060] In an exemplary embodiment, in combination with FIG. 1 and as shown in FIG. 2 , a data processing method for a key-value storage system is provided, which is described by taking the method applied to the computing node 10 in FIG. 1 as an example, and includes the following steps:

[0061] Step 202, receiving a data processing command.

[0062] Among them, the data processing command is used to instruct the computing node to process the data related to the key-value storage system, which is similar to the common Redis commands, such as write data command, access data command, delete key-value pair, etc. The distributed log service can access various real-time log data on a large scale. As a storage system, it can support data volumes far exceeding the memory. Optionally, the distributed log service uses Apache Bookkeeper, which has the following characteristics: all data is written in an appended manner, and the written data cannot be modified.

[0063] Step 204, based on the data processing command, processes at least one of the hash table data structure stored in the memory, the data stored in each log file, the cache data stored in the cache layer of the memory, the configuration data, and the metadata; wherein the hash table data structure includes multiple pairs of keys and values, and the values ​​are log information storing corresponding data.

[0064] Among them, a hash table (HASH TABLE) data structure stored in memory is constructed, including multiple key (KEY) value (VALUE) pairs, and VALUE is the log information storing the corresponding data. The log information refers to an identifier that uniquely identifies and tracks log records, and is used to determine the location of the corresponding data in the log file. Based on the log information, the computing node can access the data stored in the log file. Using the efficient HASH TABLE data structure, dynamic and high-concurrency updates and queries are supported.

[0065] The interface of the configuration center is implemented on the computing node, supporting the reading, adding, updating, and deleting of configuration data, as well as dynamic loading and updating of metadata and configuration data, etc. The configuration center can adopt ETCD or Zookeeper architecture, which is not limited in this embodiment.

[0066] Implement the abstract interface layer of distributed logs on the computing nodes, supporting the following operations: create logs, delete logs, query log metadata, randomly query logs, traverse logs in a specified range, write logs in append mode, and write logs in batches. Optionally, implement the abstract interface layer for distributed logs based on the Bookkeeper interface.

[0067] Implement a memory cache layer in the computing node and establish a memory mapping from log information to actual data. Combined with the data cache strategy, cache some data in memory and store the full data in the distributed log service. Optionally, the computing node supports statistics on cached data, such as the size, quantity, and lifetime of the data. The configuration data stored in the configuration center can be used to configure the data capacity of the cache layer, as well as a variety of data expiration strategies, such as LRU (least recently used algorithm), LFU (least frequently used algorithm), CLOCK (clock algorithm), RANDOM (random algorithm), etc. In this way, elastic expansion and contraction can be achieved on demand, effectively improving resource utilization.

[0068] In an optional implementation, all KEY data are converted into 64-bit unsigned numbers as HASH IDs using a HASH function, and a HASH TABLE data structure stores multiple HASH IDs and corresponding LOG IDs (i.e., log information). Optionally, a fixed-length linked list is used in the HASH TABLE data structure to store the values ​​corresponding to repeated HASH IDs.

[0069] It is understandable that different data processing commands target different data and have different data processing logics. This embodiment provides a corresponding implementation method for each data processing command, which can dynamically update the hash table data structure, data stored in each log file, cache data, configuration data, and metadata, thereby improving the flexibility and efficiency of data processing.

[0070] In an optional implementation, referring to FIG3 , the computing node includes the following modules: a network protocol layer, a network access layer, a command interpretation module, a timed task module, an execution layer, a metadata and configuration module, a cache layer, and a log management module. The network protocol layer is used to correctly interpret the RESP protocol (Redis serialization protocol). The network access layer uses the access control list (ACL) to implement the login verification function of Redis, and adds command-level permission checks for subsequent command execution. The command interpretation module is used to interpret commands based on the interpretation results of the network protocol layer, and convert requests into specific commands and parameters. The timed task module is used to manage various timed tasks of logs, such as: creating new log files in advance, merging files, and starting checkpoints at a scheduled time. The execution layer is used to execute commands. The metadata and configuration module is used to interact with the configuration center to realize configuration reading, adding, updating, and deleting, as well as dynamic loading and updating of metadata and configuration. The cache layer is used to store HASH TABLE data structures and cache data. The log management module is used to interact with the distributed log service and perform the following operations: create logs, delete logs, query log metadata, randomly query logs, traverse logs within a specified range, write logs in append mode, and write logs in batches.

[0071] In the data processing method of the above-mentioned key-value storage system, the key-value storage system includes a computing node, a distributed log service and a configuration center, the distributed log service is used to store data in each log file, and the configuration center is used to store configuration data and metadata of each log file; the computing node receives a data processing command; based on the data processing command, at least one of the hash table data structure stored in the memory, the data stored in each log file, the cache data stored in the cache layer of the memory, the configuration data and the metadata is processed; wherein, the hash table data structure includes multiple pairs of keys and values, and the value is the log information for storing the corresponding data. Through the above-mentioned method, based on the distributed log service as a storage system, there is no need to rely heavily on the memory, which greatly improves the upper limit of the data capacity. Based on the designed hash table data structure, the high availability of data can be guaranteed. Based on the computing nodes and the distributed log service, the separation of computing and storage is realized, the utilization rate of resources is improved, and the flexibility and efficiency of computing are improved.

[0072] In an exemplary embodiment, the data processing command includes an access data command; the method also includes: determining the corresponding first log information in the hash table data structure based on the key corresponding to the access data command; searching for the corresponding data in the cache layer based on the first log information; if the data is not found, calling the interface of the distributed log service and passing in the first log information; receiving the target data fed back by the distributed log service based on the first log information; storing the cached data in the cache layer according to the first log information and the target data, and feeding back the target data to the client that sent the access data command.

[0073] Among them, when the command execution process needs to access data, the first log information of the target (i.e., LOG ID) is found through the hash table data structure, and the corresponding data is searched in the cache layer based on the LOG ID. If the data exists, the callback function is called. If the data does not exist, the interface of the distributed log service is asynchronously called to query the corresponding data, the callback function is called, and the cache data of the cache layer is updated, and the queried target data is fed back to the client that sent the access data command. By caching the data, it is convenient to quickly read the target data later, significantly reducing the response time and improving the overall performance of the storage system. In this way, a data reading and access method for the key-value storage system is provided to improve the availability of the key-value storage system.

[0074] In an optional implementation, in combination with FIG3 and with reference to FIG4, the client sends a request, and the access data flow is executed through the network protocol layer, the command interpretation module, and the execution layer to detect whether the HASH ID corresponding to the KEY exists in the HASH TABLE. If not, the result is returned (no data is found). If it exists, it is determined whether the KEY or VALUE is in the cache. If it exists, the result (data queried from the cache) is returned. If it does not exist, the distributed log is queried, the result (data queried from the distributed log) is returned, and the VALUE cache is updated.

[0075] It can be understood that, referring to Figure 5, the KEY=HASH ID and VALUE=LOG ID stored in the HASH TABLE, assuming that the HASH ID calculated by the HASH function is 90127756092, the corresponding LOG ID is determined to be (123,43) by querying the HASH TABLE, pointing to the log data with LEGER=123 and ID=43 in the distributed log. Assuming that the HASH ID calculated by the HASH function is 83926841, the corresponding LOG ID is determined to be (781,41) by querying the HASH TABLE, pointing to the log data with LEGER=781 and ID=41 in the distributed log.

[0076] In an exemplary embodiment, the data processing command includes a write data command; the method also includes: based on the write data command, calling the interface of the distributed log service, passing in the data to be written corresponding to the write data command, so as to instruct the distributed log service to store the data to be written in an idle log file, and feedback the second log information storing the data to be written; receiving the second log information feedback by the distributed log service; adding a key-value pair in the hash table data structure according to the key corresponding to the data to be written and the second log information, and storing the cached data in the cache layer according to the second log information and the data to be written.

[0077] For the data write command, the distributed log service interface is asynchronously called to write log data. After success, the callback function is called to update the HASH TABLE information (add KEY-VALUE pairs, VALUE is the second log information) and cache data, and the write result is fed back to the client. This provides a data writing method for the key-value storage system and improves the availability of the key-value storage system.

[0078] In an optional implementation, for commands to add or modify KEY, the distributed log service interface is asynchronously called to write log data. After success, the callback function is called to update the HASH TABLE information and cached data, and the writing result is fed back to the client.

[0079] In an exemplary embodiment, the data processing command includes a delete command; the method also includes: based on the key corresponding to the delete command, determining the corresponding third log information in the hash table data structure; calling the interface of the distributed log service, passing in the third log information to instruct the distributed log service to determine the data to be deleted based on the third log information, and writing the information corresponding to the data to be deleted in the tombstone identification log; according to the key corresponding to the delete command, deleting the key-value pair in the hash table data structure, and deleting the cached data stored in the cache layer according to the third log information.

[0080] Among them, the deletion command is a delete KEY command or a delete data command. For the deletion command, the interface of the distributed log service is asynchronously called to write a special tombstone identification log. Specifically, the data to be deleted indicated by the third log information corresponding to the KEY is determined, and the information corresponding to the data to be deleted is written in the tombstone identification log. After success, the callback function is called to update the information and cached data of the HASH TABLE, and the deletion result is fed back to the client. Provide a data deletion method for the key-value storage system to improve the availability of the key-value storage system.

[0081] It is understandable that the "Tombstone Marker" strategy does not remove data from the storage system immediately, but achieves the effect of deletion by marking the data as "deleted" or "invalid". In a distributed system, data may exist in multiple copies. The tombstone log can ensure that all copies are deleted in a coordinated manner to maintain data consistency. The key-value storage system physically deletes the data indicated by the tombstone log at a later time or according to a specific strategy (such as a garbage collection strategy), which helps optimize the utilization of storage resources.

[0082] In an exemplary embodiment, the method also includes: obtaining metadata of the kth log file; the kth log file is the log file currently written by the distributed log service; if the metadata meets the preset switching conditions, calling the distributed log service to create a new k+2th log file, and using the k+1th log file as the currently written log file; synchronizing the metadata of the kth log file to the configuration center.

[0083] Wherein, k is an integer greater than or equal to 1. In conjunction with Figure 3, the scheduled task module manages various scheduled tasks of the log, including: building the next log file in advance. Specifically, the metadata (including time, data volume, number of records, etc.) of the current log file (i.e., the kth log file) is counted and saved. If the current log file triggers the configured threshold, and the metadata meets the preset switching conditions (for example, the data volume is greater than the threshold, the number of records is greater than the threshold, etc., the threshold is the configuration data stored in the configuration center), the current log file is switched to the next file (i.e., the k+1th log file), and the next unwritten log file (i.e., the k+2th log file) is created at the same time. The metadata, configuration, etc. of each log file involved in this process need to be synchronized to the configuration center.

[0084] In this embodiment, the unwritten log files are created in advance, which can reduce the waiting time for log writing, thereby improving the performance of the key-value storage system.

[0085] In an exemplary embodiment, as shown in FIG6 , the method further includes:

[0086] Step 602: Count the invalid data information and valid data information of each log file.

[0087] Among them, the computing node maintains some statistical information for each log file storing data, including invalid data information (for example, the number of invalid records, the amount of invalid data) and valid data information (for example, the number of valid records, the amount of valid data), etc.

[0088] Step 604: Determine multiple log files that meet a preset merging condition based on the invalid data information and the valid data information.

[0089] Step 606: Call the distributed log service to merge the data stored in multiple log files.

[0090] Among them, the computing node merges multiple log files that meet the preset merging conditions within the available operation and maintenance time. The preset merging conditions include the number of valid records less than the threshold, the amount of valid data less than the threshold, the number of invalid records greater than the threshold, the amount of invalid data greater than the threshold, etc. Combined with Figure 3, the scheduled task module manages various scheduled tasks of the log, including: scheduled merging of log files.

[0091] In this embodiment, by setting a policy for periodically merging log files, the number of log files of the distributed log service can be reduced, the complexity of log management can be reduced, and the utilization rate of storage resources can be optimized.

[0092] In an exemplary embodiment, step 306 includes: determining, based on a hash table data structure, fourth log information corresponding to valid data in multiple log files; calling an interface of a distributed log service, creating a new first log file, and writing the valid data corresponding to the fourth log information into the first log file; synchronizing metadata corresponding to the first log file to a configuration center, and updating the hash table data structure based on the first log file.

[0093] Among them, determine which specific data in the multiple log files to be merged is valid from the HASH TABLE, create a new first log file, traverse and batch read the old log files (the log files pointed to by the fourth log information), and rewrite them to the new first log file. Update the metadata to the configuration center, and synchronously update the index information of the HASH TABLE. At this time, if there are no other associations in the old log files, these old log files without other associations can be deleted.

[0094] In an optional implementation, BITMAP is used to save valid information of multiple log files. When reading old log files, BITMAP is used to quickly locate valid data and rewrite it to new log files, which can improve the efficiency of batch data reading and writing.

[0095] In an optional implementation, referring to FIG. 7 , the process of merging logs includes: calling the interface of the distributed log service to create a new log, and the distributed log returns the log ID; reading log files in batches, filtering invalid data, and calling the interface of the distributed log service to batch intervene in log data; updating HASH TABLE data, and updating metadata to the configuration center.

[0096] In an exemplary embodiment, the method further includes: obtaining checkpoint information from a configuration center at startup; calling an interface of a distributed log service to read log data in a log file corresponding to the checkpoint information; constructing a hash table data structure based on the log data;

[0097] After step 306, the method further includes: calling a distributed log service, creating a new second log file, and writing log index information in the second log file; generating target checkpoint information based on the metadata of the second log file, and synchronizing it to the configuration center; deleting the historical checkpoint information in the configuration center; calling a distributed log service, and deleting the log files associated with the historical checkpoint information in multiple log files.

[0098] When the computing node starts, it finds the latest checkpoint information (Checkpoint) from the configuration center, reads data from the specified log file based on the Checkpoint, and traverses the subsequent log information from the corresponding log position to construct a complete HASH TABLE. Checkpoint is equivalent to a snapshot of the key-value storage system. When the system crashes, recovers, and starts, the HASH TABLE can be quickly rebuilt from the Checkpoint.

[0099] In this embodiment, Checkpoints are stored in the configuration center. The computing center determines whether to trigger Checkpoints regularly based on the statistical information collected during operation. Optionally, Checkpoints may be triggered when the amount of new data in the storage system reaches a certain threshold; Checkpoints may also be triggered when the log file changes due to a merge operation. Combined with Figure 3, the scheduled task module manages various scheduled tasks of the log, including: regularly triggering Checkpoints.

[0100] In this case, a new log file is created to record the latest log location information. The current HASH TABLE is traversed and its index information is written to the new log file. The metadata is updated and the metadata of the Checkpoint (such as the latest log location, the status of the HASH TABLE, etc.) is updated to the configuration center. Based on configurable rules, the old Checkpoint data is cleaned up and the log files associated with the old Checkpoint are deleted synchronously. The metadata of this part of the cleanup operation is synchronized to the configuration center.

[0101] In an optional implementation, referring to FIG8 , the Checkpoint execution process includes: calling the interface of the distributed log service to create a new log, and the distributed log returns the log ID; traversing the HASH TABLE and streaming data to the log; updating the Checkpoint data and synchronizing it to the configuration center; cleaning up the old Checkpoint data and deleting the log files associated with the old Checkpoint; synchronizing the metadata corresponding to the deletion operation to the configuration center.

[0102] In this embodiment, after the log files are merged, the latest checkpoint information is generated to ensure that the system state can be saved regularly. When the system starts, the system state can be quickly restored by reading the latest checkpoint and the corresponding log file, thereby improving the reliability and recovery capability of the key-value storage system.

[0103] In an exemplary embodiment, step 202 includes: receiving a Redis command sent by a client; performing permission verification on the client using an access control list; and converting the Redis command into a data processing command after the permission verification passes.

[0104] It can be understood that, in conjunction with Figure 3, the computing node implements a Redis network protocol interpretation layer that can correctly interpret the RESP protocol; implements the network access layer, uses the access control list (ACL) to implement the Redis login verification function, and adds command-level permission checks for subsequent command execution; implements the command interpretation module to convert Redis commands into specific data processing commands and parameters. Compatible with the Redis network protocol, it can access the Redis ecosystem and various existing clients, broadening the use scenarios of Redis.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] In an exemplary embodiment, as shown in Figure 1, an embodiment of the present application provides a key-value storage system, which includes a computing node 10, a distributed log service 20 and a configuration center 30. The distributed log service 20 is used to store data in each log file, and the configuration center 30 is used to store configuration data and metadata of each log file; when the computing node 10 executes a computer program, the steps of the method provided in any embodiment of the present application are implemented.

[0107] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: receiving a data processing command; based on the data processing command, processing at least one of a hash table data structure stored in the memory, data stored in each log file, cache data stored in the cache layer of the memory, configuration data, and metadata; wherein the hash table data structure includes multiple pairs of keys and values, and the values ​​are log information storing corresponding data.

[0108] In the above key-value storage system, the distributed log service is used as the storage system, which does not need to be highly dependent on memory, greatly increasing the upper limit of data capacity. The designed hash table data structure can ensure high data availability. Based on computing nodes and distributed log services, computing and storage are separated, resource utilization is improved, and computing flexibility and efficiency are improved.

[0109] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: based on the key corresponding to the access data command, determine the corresponding first log information in the hash table data structure; based on the first log information, search for the corresponding data in the cache layer; if the data is not found, call the interface of the distributed log service and pass in the first log information; receive the target data fed back by the distributed log service based on the first log information; based on the first log information and the target data, store the cached data in the cache layer, and feed back the target data to the client that sent the access data command.

[0110] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: based on a write data command, calling the interface of the distributed log service, passing in the data to be written corresponding to the write data command, to instruct the distributed log service to store the data to be written in an idle log file, and feedback the second log information storing the data to be written; receiving the second log information feedback by the distributed log service; adding a key-value pair to the hash table data structure according to the key corresponding to the data to be written and the second log information, and storing the cached data in the cache layer according to the second log information and the data to be written.

[0111] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: based on the key corresponding to the delete command, determining the corresponding third log information in the hash table data structure; calling the interface of the distributed log service, passing in the third log information to instruct the distributed log service to determine the data to be deleted based on the third log information, and write the information corresponding to the data to be deleted in the tombstone identification log; according to the key corresponding to the delete command, deleting the key-value pair in the hash table data structure, and deleting the cached data stored in the cache layer according to the third log information.

[0112] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: obtaining metadata of the kth log file; the kth log file is the log file currently written by the distributed log service; if the metadata meets the preset switching conditions, calling the distributed log service to create a new k+2th log file, and using the k+1th log file as the currently written log file; synchronizing the metadata of the kth log file to the configuration center.

[0113] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: counting the invalid data information and valid data information of each log file; determining multiple log files that meet preset merging conditions based on the invalid data information and the valid data information; calling a distributed log service to merge the data stored in multiple log files.

[0114] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: determining, based on a hash table data structure, fourth log information corresponding to valid data in multiple log files; calling an interface of a distributed log service, creating a new first log file, and writing valid data corresponding to the fourth log information into the first log file; synchronizing metadata corresponding to the first log file to a configuration center, and updating the hash table data structure based on the first log file.

[0115] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: at startup, obtain checkpoint information from the configuration center; call the interface of the distributed log service to read the log data in the log file corresponding to the checkpoint information; build a hash table data structure based on the log data; call the distributed log service to merge the data stored in multiple log files, call the distributed log service to create a second log file, and write log index information in the second log file; generate target checkpoint information based on the metadata of the second log file, and synchronize it to the configuration center; delete the historical checkpoint information in the configuration center; call the distributed log service to delete the log files associated with the historical checkpoint information in multiple log files.

[0116] In an exemplary embodiment, the computing node 10 implements the following steps when executing a computer program: receiving a Redis command sent by a client; performing authority verification on the client using an access control list; and converting the Redis command into a data processing command after the authority verification passes.

[0117] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method for a key-value storage system, characterized in that: The key-value storage system includes computing nodes, a distributed log service and a configuration center, wherein the distributed log service is used to store data in each log file, and the configuration center is used to store configuration data and metadata of each log file; The method is applied to the computing node, comprising: receiving data processing commands; Based on the data processing command, a hash table data structure stored in the memory, data stored in each log file, cache data stored in the cache layer of the memory, the configuration data, and at least one of the metadata are processed; wherein the hash table data structure includes multiple pairs of keys and values, and the values ​​are log information storing corresponding data.

2. The method according to claim 1, characterized in that The data processing command includes an access data command; the method further includes: Based on the key corresponding to the access data command, determining corresponding first log information in the hash table data structure; Based on the first log information, searching for corresponding data in the cache layer; If no data is found, calling the interface of the distributed log service and passing in the first log information; Receiving target data fed back by the distributed log service based on the first log information; According to the first log information and the target data, the cache data is stored in the cache layer, and the target data is fed back to the client that sends the access data command.

3. The method according to claim 1, characterized in that The data processing command includes a write data command; the method further includes: Based on the write data command, call the interface of the distributed log service, pass in the data to be written corresponding to the write data command, so as to instruct the distributed log service to store the data to be written in an idle log file, and feed back the second log information storing the data to be written; Receive the second log information fed back by the distributed log service; According to the key corresponding to the data to be written and the second log information, a key-value pair is added to the hash table data structure, and according to the second log information and the data to be written, cache data is stored in the cache layer.

4. The method according to claim 1, characterized in that The data processing command includes a delete command; the method further includes: Based on the key corresponding to the delete command, determining corresponding third log information in the hash table data structure; Calling the interface of the distributed log service, passing in the third log information, so as to instruct the distributed log service to determine the data to be deleted based on the third log information, and writing information corresponding to the data to be deleted in the tombstone identification log; According to the key corresponding to the delete command, the key-value pair is deleted in the hash table data structure, and according to the third log information, the cache data stored in the cache layer is deleted.

5. The method according to claim 1, characterized in that The method further comprises: Obtain metadata of the kth log file; the kth log file is the log file currently written by the distributed log service; If the metadata meets the preset switching condition, the distributed log service is called to create a new k+2th log file, and the k+1th log file is used as the currently written log file; Synchronize the metadata of the kth log file to the configuration center.

6. The method according to claim 1, characterized in that The method further comprises: Count the invalid data information and valid data information of each log file; Determining, according to the invalid data information and the valid data information, a plurality of log files that meet a preset merging condition; The distributed log service is called to merge the data stored in the multiple log files.

7. The method according to claim 6, characterized in that The calling of the distributed log service to merge the data stored in the multiple log files includes: Determine, according to the hash table data structure, fourth log information corresponding to the valid data in the plurality of log files; Calling the interface of the distributed log service, creating a new first log file, and writing the valid data corresponding to the fourth log information into the first log file; The metadata corresponding to the first log file is synchronized to the configuration center, and the hash table data structure is updated according to the first log file.

8. The method according to claim 6, characterized in that The method further comprises: At startup, obtaining checkpoint information from the configuration center; Calling the interface of the distributed log service to read the log data in the log file corresponding to the checkpoint information; Constructing a hash table data structure according to the log data; After calling the distributed log service and merging the data stored in the multiple log files, the method further includes: Calling the distributed log service, creating a second log file, and writing log index information into the second log file; Generate target checkpoint information according to the metadata of the second log file, and synchronize it to the configuration center; Deleting historical checkpoint information in the configuration center; The distributed log service is called to delete the log files associated with the historical checkpoint information from the multiple log files.

9. The method according to any one of claims 1 to 8, characterized in that The receiving of the data processing command comprises: Receive Redis commands sent by the client; Using an access control list to verify the permissions of the client; After the authority verification is passed, the Redis command is converted into a data processing command.

10. A key-value storage system, characterized in that: The key-value storage system includes computing nodes, a distributed log service and a configuration center, wherein the distributed log service is used to store data in each log file, and the configuration center is used to store configuration data and metadata of each log file; when the computing node executes a computer program, the steps of the method described in any one of claims 1 to 9 are implemented.

Images