Data processing method, device, system and equipment and storage medium

Abstract

The invention discloses a data processing method, device and system, equipment and a storage medium, and relates to the technical field of computers. When the data processing request is received, the server does not need to perform hash operation on the target key, and the identifier of the target hash bucket where the target key is located can be determined directly based on the target hash value included in the data processing request, so that the operation corresponding to the to-be-processed data is executed for the target hash bucket, and the data processing efficiency can be improved. The method comprises the following steps: receiving a data processing request from a client, wherein the data processing request comprises a target hash value of a target key and an identifier of a target partition to which the target key belongs; based on the target hash value, determining an identifier of a target fragment to which the target key belongs from at least two fragments included in the target partition; determining an identifier of a target hash bucket where the target key is located from a hash table corresponding to the target fragment based on the target hash value; and aiming at the target hash bucket, executing an operation corresponding to the to-be-processed data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a data processing method, apparatus, system, device, and storage medium. Background Art

[0002] Currently, data can be stored in the form of key-value pairs. When performing related operations (such as updating, deleting, etc.) on the stored data, the server can first perform certain processing on the key of the data, and then execute the processing process for the data.

[0003] However, in the above method, since the server processes the key of the data more times, and the multiple processes of the key consume a large amount of computing / processing resources, thus affecting the efficiency of data processing. Summary of the Invention

[0004] This application provides a data processing method, apparatus, system, device, and storage medium, which solves the technical problem in the related art that the key of the data is processed more times, thus affecting the efficiency of data processing.

[0005] In a first aspect, a data processing method is provided, which is applied to an in-memory key-value database. The in-memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a hash table. The method includes: after receiving a data processing request from a client, determining an identifier of a target shard to which a target key belongs and an identifier of a target hash bucket where the target key is located from at least two shards included in a target partition to which the target key belongs based on a target hash value of the target key included in the data processing request; and then performing an operation corresponding to data to be processed on the target hash bucket.

[0006] It can be understood that the method described in the first aspect can be executed by a server, and the server can be installed in a computing device, which can be a server, a device including a server, or a chip in a server.

[0007] In this application, since the data processing request received by the server includes the target hash value of the target key, the server does not need to perform a hash operation on the target key, and can directly determine the identifier of the target shard to which the target key belongs and the identifier of the target hash bucket where the target key is located based on the target hash value, and then perform an operation corresponding to the data to be processed on the target hash bucket, which can improve the efficiency of data processing.

[0008] In a possible design, the specific method for determining the identifier of the target partition to which the target key belongs includes: performing a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard. The specific method for determining the identifier of the target hash bucket where the target key is located includes: performing a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard to obtain the identifier of the target hash bucket.

[0009] Based on this possible design, by performing a modulo operation on the target hash value according to the number of shards included in the target partition to which the target key belongs and performing a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard, the client can conveniently and quickly determine the identifier of the target hash bucket where the target key is located based on two simple division operations (i.e., modulo operations). Thus, the operation corresponding to the data to be processed can be executed for this target hash bucket, and the efficiency of data processing of the data to be processed can be improved.

[0010] In a possible design, the shards in a partition are uniformly numbered.

[0011] Based on this possible design, since the shards in a partition are uniformly numbered, that is, the identifiers of the shards included in the same partition do not repeat. In this way, the server can determine the identifier of the target shard to which the target key belongs uniquely from at least two shards included in the target partition to which the target key belongs, and thus implement the processing process of the data to be processed based on this unique target shard, which can improve the accuracy of data processing.

[0012] In a possible design, for the target hash bucket, performing the operation corresponding to the data to be processed specifically includes: parsing and processing the data processing request to obtain the type of this data processing request; for the target hash bucket, performing the operation corresponding to the data to be processed based on the type of this data processing request.

[0013] Based on this possible design, since the processing priorities of different types of data processing requests are different. For example, the processing priority of I / O requests is higher than that of non-I / O requests. In this way, the server can determine the priority of the data processing request based on the type of the data processing request, and thus preferentially process the data processing request with a higher priority / importance level.

[0014] In a possible design, the above-mentioned parsing and processing of the data processing request to obtain the type of this data processing request specifically includes: controlling the target I / O thread to parse and process the data processing request to obtain the type of this data processing request; where the target I / O thread corresponds to the client.

[0015] Based on this possible design, the server can parse and process the data processing request by controlling the target I / O thread, which can save the processing resources of the main thread in the server, avoid putting too much pressure on the main thread, and improve the availability of the threads included in the server.

[0016] In a possible design, for the above-mentioned target hash bucket, performing an operation on the data to be processed based on the type of the data processing request specifically includes: when the type of the data processing request includes an I / O request, controlling the target I / O thread to perform an operation on the data to be processed based on the type of the data processing request for the target hash bucket; or when the type of the data processing request includes a non-I / O request, controlling the worker thread to perform an operation on the data to be processed based on the type of the data processing request for the target hash bucket.

[0017] Based on this possible design, when the type of the data processing request includes an I / O request, it indicates that the priority of this data processing request is relatively high, and the server can control the target I / O thread to process this data processing request. When the type of the data processing request includes a non-I / O request, it indicates that the priority of this data processing request is relatively low, and the server can control the worker thread to process this data processing request. That is, the server can determine the priority / importance level of the data processing request based on the type of the data processing request, so as to control the target I / O thread to process requests / tasks with a higher importance level, and control the worker thread to process requests / tasks with a lower importance level, which can improve the efficiency of data processing.

[0018] In a possible design, the above-mentioned non-I / O requests include: KEYS, FLUSHALL, or bigkey.

[0019] Based on this possible design, since the non-I / O requests include KEYS, FLUSHALL, or bigkey, when the type of the data processing request includes a non-I / O request, it indicates that the priority of this data processing request is relatively low (or the processing efficiency of this data processing request is relatively low). At this time, the server can allocate this data processing request to the worker thread and control the worker thread to process this data processing request. This can avoid blocking the I / O thread caused by requests / tasks with a lower processing efficiency, ensure that the I / O thread can respond to requests / tasks with a higher priority / processing efficiency in a timely manner, and improve the stability of the data processing system.

[0020] In a possible design, for the above-mentioned target hash bucket, performing an operation corresponding to the data to be processed based on the type of the data processing request specifically includes: estimating the processing duration for executing the data processing request based on the type of the data processing request; when the processing duration is greater than or equal to the duration threshold, controlling a worker thread to perform an operation corresponding to the data to be processed for the target hash bucket; or when the processing duration is less than the duration threshold, controlling a target I / O thread to perform an operation corresponding to the data to be processed for the target hash bucket.

[0021] Based on this possible design, by estimating the processing duration for executing a data processing request based on the type of the data processing request, the server can control the worker thread to process requests / tasks that require a longer processing duration, and control the target I / O thread to process requests / tasks that require a shorter processing duration. While improving the data processing efficiency, it can avoid the long-term occupation of the I / O thread and improve the processing performance of the I / O thread.

[0022] In a possible design, for the above-mentioned target hash bucket, performing an operation corresponding to the data to be processed based on the type of the data processing request specifically includes: estimating the data volume targeted by the data processing request based on the type of the data processing request; when the data volume is greater than or equal to the preset data volume, controlling a worker thread to perform an operation corresponding to the data to be processed for the target hash bucket; or when the data volume is less than the preset data volume, controlling a target I / O thread to perform an operation corresponding to the data to be processed for the target hash bucket.

[0023] Based on this possible design, by estimating the data volume targeted by the data processing request based on the type of the data processing request, the server can control the worker thread to process requests / tasks with a larger targeted data volume, and control the target I / O thread to process requests / tasks with a smaller targeted data volume. It can avoid blocking the I / O thread due to requests / tasks with a larger data volume and ensure that the server (or the I / O thread) can respond to requests / tasks with a smaller data volume in a timely manner.

[0024] In a possible design, the above-mentioned controlling a worker thread to perform an operation corresponding to the data to be processed for the target hash bucket includes: adding the data processing request to the task queue corresponding to the worker thread; controlling the worker thread to process the data processing request in the task queue for the target hash bucket to perform an operation corresponding to the data to be processed.

[0025] Based on this possible design, by adding the data processing request to the task queue corresponding to the worker thread, the server can, when controlling the worker thread to process relevant requests / tasks, process the tasks (including data processing requests) in the task queue in sequence, and can ensure the timing of data processing.

[0026] In a possible design, the number of shards included in a partition is the maximum capacity value of the key space of the in-memory key-value database.

[0027] Based on this possible design, since the number of components included in a partition is the maximum capacity value of the key space of the in-memory key-value database, in this in-memory key-value database, the key space / data space can be fully partitioned, while improving the memory usage efficiency, it can meet the multi-concurrency scenario to support the data interaction and processing process between multiple clients and multiple servers.

[0028] In a second aspect, a data processing method is provided, which is applied to an in-memory key-value database. The in-memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a server. The method includes: after obtaining the target hash value of the target key, determining the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value; and then sending a data processing request to the server corresponding to the target shard to request the server to perform an operation corresponding to the data to be processed, where the data processing request includes the target hash value and the identifier of the target partition.

[0029] It can be understood that the method described in the second aspect can be executed by a client. The client can be installed in a computing device, and the computing device can be a terminal, a device including a terminal, or a chip in the terminal.

[0030] In this application, after the client obtains the target hash value of the target key, it can determine the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value. Since one shard corresponds to one node / server, in this way, the client can accurately determine the server corresponding to the target shard based on the identifier of the target shard, and then send a data processing request to the server to request the server to perform an operation corresponding to the data to be processed. It can conveniently and quickly determine the server storing the data to be processed based on the identifier of the target shard to which the target key belongs, thereby improving the data processing efficiency.

[0031] In a possible design, the specific way to determine the identifier of the target partition to which the target key belongs includes: performing a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

[0032] Based on this possible design, by performing a modulo operation on the target hash value according to the number of shards included in the target partition to which the target key belongs, the client can conveniently and quickly determine the identifier of the target shard to which the target key belongs based on a simple division operation (i.e., modulo operation), thereby improving the data processing efficiency.

[0033] In a possible design, before sending a data processing request to a server, the method further includes: determining a target shard array to which the target shard belongs based on an identifier of the target shard, where the target shard array includes identifiers of at least two consecutive shards; determining a service group corresponding to the target shard array from an array-service relationship based on the target shard array, where the array-service relationship is used to indicate shard arrays corresponding to one or more service groups; and determining a master node included in the service group corresponding to the target shard array as the server corresponding to the target shard.

[0034] Based on this possible design, the client can determine the target shard array to which the target shard belongs and the service group corresponding to the target shard array based on the identifier of the target shard, and thus determine the master node included in the service group as the server corresponding to the target shard. It is possible to conveniently and quickly determine the server corresponding to the target shard based on the identifier of the target shard, which can improve the efficiency of data processing.

[0035] In a third aspect, a data processing method is provided, which is applied to an in-memory key-value database. The method includes: receiving a data processing request from a client, where the data processing request includes a target key of data to be processed; determining a target hash slot to which the target key belongs based on the target key and determining a target hash bucket in which the target key is located from the target hash slot based on the target key; parsing and processing the data processing request to obtain the type of the data processing request; and performing an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0036] It can be understood that the method described in the third aspect can be executed by a server, and the server can be installed in a computing device, which can be a server, a device including a server, or a chip in the server.

[0037] In this application, after receiving a data processing request from a client, the server can respectively determine a target hash slot to which the target key belongs and a target hash bucket in which the target key is located based on the target key included in the data processing request. Then, for the target hash bucket, an operation corresponding to the data to be processed is performed based on the type of the data processing request. Since the processing priorities of different types of data processing requests are different, for example, the processing priority of an I / O request is higher than that of a non-I / O request, the server can thus determine the priority of the data processing request based on the type of the data processing request, and thus preferentially process data processing requests with a higher priority / importance level.

[0038] In a possible design, the above-mentioned parsing and processing of the data processing request to obtain the type of the data processing request specifically includes: controlling the target I / O thread to parse and process the data processing request to obtain the type of the data processing request; wherein, the target I / O thread corresponds to the client.

[0039] In a possible design, the above-mentioned operation corresponding to the data to be processed is performed on the target hash bucket based on the type of the data processing request, specifically including: when the type of the data processing request includes an I / O request, controlling the target I / O thread to perform the operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request; or when the type of the data processing request includes a non-I / O request, controlling the worker thread to perform the operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0040] In a possible design, the above-mentioned non-I / O requests include: KEYS, FLUSHALL, or bigkey.

[0041] In a possible design, the above-mentioned operation corresponding to the data to be processed is performed on the target hash bucket based on the type of the data processing request, specifically including: estimating the processing duration of executing the data processing request based on the type of the data processing request; when the processing duration is greater than or equal to the duration threshold, controlling the worker thread to perform the operation corresponding to the data to be processed on the target hash bucket; or when the processing duration is less than the duration threshold, controlling the target I / O thread to perform the operation corresponding to the data to be processed on the target hash bucket.

[0042] In a possible design, the above-mentioned operation corresponding to the data to be processed is performed on the target hash bucket based on the type of the data processing request, specifically including: estimating the data volume targeted by the data processing request based on the type of the data processing request; when the data volume is greater than or equal to the preset data volume, controlling the worker thread to perform the operation corresponding to the data to be processed on the target hash bucket; or when the data volume is less than the preset data volume, controlling the target I / O thread to perform the operation corresponding to the data to be processed on the target hash bucket.

[0043] In a possible design, the above-mentioned controlling the worker thread to perform the operation corresponding to the data to be processed on the target hash bucket includes: adding the data processing request to the task queue corresponding to the worker thread; controlling the worker thread to process the data processing request in the task queue for the target hash bucket to perform the operation corresponding to the data to be processed.

[0044] In addition, the technical effects in the possible designs included in the data processing method described in the third aspect can refer to the technical effects in the possible designs corresponding to the first aspect above, which will not be elaborated here.

[0045] In a fourth aspect, a data processing apparatus is provided, which is applied to an in-memory key-value database. The in-memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a hash table. The apparatus includes a transceiver module, a determination module, and a processing module; the transceiver module is configured to receive a data processing request from a client, the data processing request includes a target hash value of a target key and an identifier of a target partition to which the target key belongs; the determination module is configured to determine, based on the target hash value, an identifier of a target shard to which the target key belongs from at least two shards included in the target partition; the determination module is further configured to determine, based on the target hash value, an identifier of a target hash bucket where the target key is located from the hash table corresponding to the target shard; the processing module is configured to perform an operation corresponding to the data to be processed on the target hash bucket.

[0046] In a possible design, the processing module is specifically configured to perform a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard; the processing module is further specifically configured to perform a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard to obtain the identifier of the target hash bucket.

[0047] In a possible design, the shards in a partition are uniformly numbered.

[0048] In a possible design, the processing module is specifically configured to parse and process the data processing request to obtain the type of the data processing request; the processing module is further specifically configured to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0049] In a possible design, the processing module is further specifically configured to control a target I / O thread to parse and process the data processing request to obtain the type of the data processing request; wherein, the target I / O thread corresponds to the client.

[0050] In a possible design, the processing module is further specifically configured to, when the type of the data processing request includes an I / O request, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request; or, the processing module is further specifically configured to, when the type of the data processing request includes a non-I / O request, control a worker thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0051] In a possible design, the above non-I / O requests include: KEYS, FLUSHALL, or bigkey.

[0052] In a possible design, the processing module is further specifically configured to estimate the processing duration for executing the data processing request based on the type of the data processing request; the processing module is further specifically configured to, when the processing duration is greater than or equal to the duration threshold, control the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket; or, the processing module is further specifically configured to, when the processing duration is less than the duration threshold, control the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, where the target I / O thread corresponds to the client.

[0053] In a possible design, the processing module is further specifically configured to estimate the data volume for which the data processing request is directed based on the type of the data processing request; the processing module is further specifically configured to, when the data volume is greater than or equal to the preset data volume, control the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket; or, the processing module is further specifically configured to, when the data volume is less than the preset data volume, control the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, where the target I / O thread corresponds to the client.

[0054] In a possible design, the processing module is further specifically configured to add the data processing request to the task queue corresponding to the worker thread; the processing module is further specifically configured to control the worker thread to process the data processing request in the task queue for the target hash bucket to perform the operation corresponding to the to-be-processed data.

[0055] In addition, the technical effects of the data processing device described in the fourth aspect can refer to the technical effects of the method described in the first aspect, which will not be elaborated here.

[0056] In a fifth aspect, a data processing device is provided, which is applied to an in-memory key-value database. The in-memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a server. The device includes a transceiver module and a determination module. The transceiver module is configured to obtain the target hash value of the target key; the determination module is configured to determine the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value; the transceiver module is further configured to send a data processing request to the server corresponding to the target shard, where the data processing request includes the target hash value and the identifier of the target partition, and the data processing request is used to request the server to perform the operation corresponding to the to-be-processed data.

[0057] In a possible design, the above data processing device further includes a processing module, and the processing module is configured to perform a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

[0058] In a possible design, the determination module is further configured to determine the target shard array to which the target shard belongs based on the identifier of the target shard, where the target shard array includes the identifiers of at least two consecutive shards; the determination module is further configured to determine the service group corresponding to the target shard array from the array service relationship based on the target shard array, where the array service relationship is used to indicate the shard arrays corresponding to one or more service groups; the determination module is further configured to determine the primary node included in the service group corresponding to the target shard array as the server corresponding to the target shard.

[0059] In addition, the technical effects of the data processing device described in the fifth aspect can refer to the technical effects of the method described in the second aspect, which will not be elaborated here.

[0060] In a sixth aspect, a data processing system is provided. The data processing system includes a client and a server. The server is configured to execute the method described in the first aspect above, and the client is configured to execute the method described in the second aspect above.

[0061] In addition, the technical effects of the data processing system described in the sixth aspect can refer to the technical effects of the methods described in the first and second aspects, which will not be elaborated here.

[0062] In a seventh aspect, a computing device cluster is provided, including at least one computing device. Each computing device includes a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or executes the method described in the second aspect.

[0063] In an eighth aspect, a computer program product including instructions is provided. When the instructions are run by a computing device cluster, the computing device cluster is caused to execute the method described in the first aspect or execute the method described in the second aspect.

[0064] In a ninth aspect, a computer-readable storage medium is provided, including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or executes the method described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Shows a partition diagram included in an in-memory key-value database in a single-machine mode provided by an embodiment of the present application;

[0066] Figure 2 Shows a partition diagram included in an in-memory key-value database in a cluster mode provided by an embodiment of the present application;

[0067] Figure 3Shows a schematic diagram of the principle for a client to find the hash bucket of a key from the hash slots of a slave node provided by an embodiment of the present application;

[0068] Figure 4 Shows a schematic diagram of the network architecture of a data processing system in a single-machine mode provided by an embodiment of the present application;

[0069] Figure 5 Shows a schematic diagram of the network architecture of a data processing system in a cluster mode provided by an embodiment of the present application;

[0070] Figure 6 Shows a schematic diagram of the hardware structure of a computing device provided by an embodiment of the present application;

[0071] Figure 7 Shows a schematic diagram of the process of a data processing method provided by an embodiment of the present application;

[0072] Figure 8 Shows a schematic diagram of partitioning and sharding in an in-memory key-value database in a single-machine mode provided by an embodiment of the present application;

[0073] Figure 9 Shows a schematic diagram of the scenario for performing an operation on data to be processed in a single-machine mode provided by an embodiment of the present application;

[0074] Figure 10 Shows a schematic diagram of the process of another data processing method provided by an embodiment of the present application;

[0075] Figure 11 Shows a schematic diagram of partitioning and sharding in an in-memory key-value database in a cluster mode provided by an embodiment of the present application;

[0076] Figure 12 Shows a schematic diagram of the principle for a client to find a target hash bucket from the hash table corresponding to a target shard provided by an embodiment of the present application;

[0077] Figure 13 Shows a schematic diagram of the process of yet another data processing method provided by an embodiment of the present application;

[0078] Figure 14 Shows a schematic diagram of the process for an I / O thread to perform an operation on data to be processed provided by an embodiment of the present application;

[0079] Figure 15 Shows a schematic diagram of another scenario for performing an operation on data to be processed in a single-machine mode provided by an embodiment of the present application;

[0080] Figure 16 Shows a schematic diagram of the process for a worker thread to perform an operation on data to be processed provided by an embodiment of the present application;

[0081] Figure 17 Shows a schematic flow diagram of adding a shared lock in a shard provided by an embodiment of the present application;

[0082] Figure 18 Shows a schematic internal structure diagram of a data processing device provided by an embodiment of the present application;

[0083] Figure 19 Shows a schematic internal structure diagram of another data processing device provided by an embodiment of the present application;

[0084] Figure 20 Shows a schematic structure diagram of a computing device cluster provided by an embodiment of the present application;

[0085] Figure 21 Shows a schematic network architecture diagram of a computing device cluster provided by an embodiment of the present application. Detailed implementation manners

[0086] To facilitate understanding of the solution provided by the embodiment of the present application, before introducing the solution provided by the embodiment of the present application, some concepts related to the present application are explained.

[0087] A database system (DBS) is a comprehensive computer system for storing, managing, and accessing data. It consists of hardware and software. The hardware includes computing devices and storage devices, and the software part includes a database (DB), a database management system (DBMS), an operating system, and application development tools supporting multiple languages, etc.

[0088] A database instance is the running instance of the DBMS in memory. It contains a set of background processes and a shared memory area for managing and operating on the data in the database.

[0089] An in-memory key-value database is a database system that stores data in memory for fast access and operation of data. In an in-memory key-value database, data is stored in the form of key-value pairs. For any data stored in the in-memory key-value database, the value of the data can be found through the key of the data.

[0090] A key is a part of a key-value pair, used to identify and access the value associated with the key, and is the unique identifier for storing and retrieving data. A key can be understood as a label or index, usually in the form of a string, which can quickly locate and access the stored data. In a in-memory key-value database, the uniqueness of the key ensures that each key corresponds to a specific value to distinguish different data.

[0091] A value is the other part of a key-value pair, associated with the key, and is the specific content where data is to be stored in a in-memory key-value database. Among them, the types of values can include various ones, such as strings, hashes, lists, and sets, etc.

[0092] A partition is to divide the data in the DBS into multiple parts according to certain rules, and each part is called a partition. Each partition can be stored and managed separately, thus achieving more efficient data management and retrieval. For example, in Redis, partitioning can be based on different types of data (such as user data, product data) or access frequencies.

[0093] Sharding is to split the data in the DBS across multiple database instances. The main purpose of sharding is to distribute the load and improve the concurrency and throughput of the system.

[0094] A hash slot is a mechanism used to achieve distributed storage and automatic sharding of data. Redis divides the entire data space (or key space) into 16384 hash slots through hash slots. After each key calculates its hash value through the cyclic redundancy check 16 (CRC16) algorithm, it takes the modulus of 16384 to determine the hash slot it belongs to, and thus maps to the corresponding node.

[0095] A hash bucket is a basic unit in the hash table data structure, used to store data / data records. A hash bucket is a data structure in the hash table used to solve hash conflicts. When multiple keys are mapped to the same hash value through the hash function, these keys (or key-value pairs) will be stored in the same hash bucket.

[0096] The following, in combination with Figure 1 and Figure 2 description, the concepts of partitioning and sharding in current Redis:

[0097] In Redis, in the single-machine mode, the in-memory key-value database includes a database instance, and at least two partitions are deployed on this database instance. As Figure 1 shown, the database instance 101 includes partition 1011 and partition 1012.

[0098] In Redis, in the cluster mode, the in-memory key-value database includes at least two database instances, and at least two partitions are deployed on each database instance. Among them, a database instance can be called a node, and a database instance corresponds to a shard. For example, as Figure 2 shown, the in-memory key-value database includes node 21 and node 22. Partition 201 and partition 211 are included in node 21, and partition 202 and partition 212 are included in node 22.

[0099] For the in-memory key-value database in the cluster mode (such as remote dictionary server (Redis) and Dragonfly), the key space of the cluster is divided into 16,384 hash slots. Each node (specifically the master node) in the cluster is responsible for handling a subset of the 16,384 hash slots. When the cluster is not undergoing reconstruction (specifically, no hash slots are being moved from one node to another node), the cluster is in a stable state. At this time, a single hash slot will be served by a single node.

[0100] Exemplarily, as Figure 3 shown, in the cluster mode, the in-memory key-value database includes 3 nodes, which are respectively labeled as: node 302, node 303, and node 304. The 3 nodes correspond to hash slot intervals (or hash slot arrays) of hash slots 0 - 5500, 5501 - 1100, and 11001 - 16383 respectively. Among them, a hash slot interval / hash slot array includes multiple hash slots.

[0101] Specifically, when the client needs to perform a processing operation (such as a read operation or a write operation) on a key-value pair, after calculating the hash value of the key in the key-value pair based on the CRC16 algorithm and taking the modulus of 16,384, it can determine the hash slot where the key is located, corresponding to Figure 3 HS = CRC16(key) % 16384 in, where HS represents the number of the hash slot where the key is located. Then the client can determine the node corresponding to the hash slot where the key is located based on the relationship between the hash slot interval / and the nodes, corresponding to Figure 3 Node = Array[HS] in.

[0102] Assume that the hash slot where the key is located is hash slot 5500. Then the client determines that the node corresponding to the above hash slot is node 302. At this time, the client sends a processing command related to the key to node 302. After receiving the processing command, node 302 performs a hash operation on the key (corresponding to Figure 3 Hash(key) in) and a modulus operation, and from the multiple hash buckets included in node 302 (corresponding to Figure 3Find the hash bucket storing the key in the Buckets), and process the data related to the key in this hash bucket.

[0103] It should be understood that in Figure 3 In the example of, the key needs to be hashed twice. The first hash is to find the hash slot where the key is located, and then determine the node corresponding to this hash slot, so as to send the relevant processing command to this node. The second hash is to find the hash bucket storing the key. And, the hash algorithms used for these two hashes are not the same.

[0104] In addition, the key space of the in-memory key-value database in the cluster mode is divided into 16,384 hash slots, and multiple hash slots can be deployed on one node. However, since too many nodes will result in too few hash slots assigned to each node, increasing the complexity of data migration and management, and also increasing the maintenance cost and network communication overhead of the cluster. Therefore, the number of nodes will be limited. Since one node corresponds to one shard, the number of shards is also limited.

[0105] Based on this, the embodiments of the present application provide a data processing method, device, system and storage medium, which are applicable to the in-memory key-value database, and are applicable to the single-machine mode and the cluster mode. The solution includes: dividing the partitions in the in-memory key-value database into at least two shards, and each shard corresponds to a hash table. In this way, after the server receives the data processing request sent by the client, in the single-machine mode, through the method of partition + shard for addressing, the identifier of the target shard to which the target key belongs can be obtained; in the cluster mode, through the method of node + partition + shard for addressing, the identifier of the target shard to which the target key belongs can be obtained. Further, the server does not need to perform a hash operation on the target key, and can directly determine the identifier of the target hash bucket where the target key is located based on the target hash value included in the data processing request, so as to execute the operation corresponding to the data to be processed for this target hash bucket, which can improve the efficiency of data processing.

[0106] The embodiments of the present application provide a data processing system, which includes at least one client and an in-memory key-value database. Among them, the in-memory key-value database can be in the single-machine mode or the cluster mode.

[0107] As Figure 4 shown, in the single-machine mode, the data processing system includes at least one client (for example, 2 clients, including client 401 and client 402) and one node, that is, node 403. Usually, in actual applications, the connections between the above devices can be wireless connections. For the convenience of intuitively representing the connection relationships between the devices, Figure 4 solid lines are used for illustration in.

[0108] Among them, the client (including client 401 and client 402) can send messages to node 403 and can also receive messages sent by node 403. For example, the client can send a data reading request to node 403, and the data reading request includes a key. The client can also receive a data reading response sent by node 403, and the data reading response includes the value corresponding to the key.

[0109] A node (such as node 403) is an entity, record, or data storage unit in a memory key-value database. It can be a physical node or a logical virtual node. When the node is a physical node, it can specifically be a server or a computing device. In the single-machine mode, all data runs on the same node, and all data processing processes are completed by one node / server / computing device.

[0110] As Figure 5 shown, in the cluster mode, the data processing system includes at least one client (such as 3 clients, including client 501, client 502, and client 503) and at least two nodes (such as 3 nodes, including node 504, node 505, and node 506). Usually, in practical applications, the connections between the above devices can be wireless connections. For the convenience of intuitively representing the connection relationships between the devices, Figure 5 solid lines are used for illustration in

[0111] Among them, the client (including client 501, client 502, and client 503) can send messages to the nodes (including node 504, node 505, and node 506) and can also receive messages sent by the nodes. For example, the client can send a data writing request to the nodes, and the data writing request includes a key and a value. The client can also receive a data writing response sent by the nodes, and the data writing response is used to notify that the key-value pair composed of the key and the value is successfully written.

[0112] A cluster / cluster mode means that multiple nodes (nodes) are combined together to form a whole and jointly provide services. These nodes can be physical machines, virtual machines, or containers, etc. They communicate and cooperate through a network to achieve high availability, load balancing, and scalability. In a memory key-value database, data can be distributed to each node. By performing data redundancy and load balancing across multiple nodes, the scalability, availability, and fault tolerance of the database can be enhanced. In the cluster / cluster mode, each node can store a part of the data, and each node has a unique identifier, an Internet Protocol (IP) address, and a port.

[0113] The above Figure 4 node 403 in Figure 5The nodes 504, 505, and 506 in can also be understood as primary / master nodes. The primary node is responsible for processing a subset of the key space in the in-memory key-value database, managing the data part corresponding to the primary node, and handling read and write requests within the primary node to ensure data consistency and availability. When data changes, such as when a write operation is performed, the primary node automatically synchronizes these changes to the replica nodes / slave nodes to achieve data redundancy and backup.

[0114] It can be understood that Figure 4 the 2 clients shown in or Figure 5 the 3 clients and 3 nodes shown in are only examples in the embodiments of the present application. The embodiments of the present application do not specifically limit the number of clients and the number of nodes included in the data processing system.

[0115] In an alternative implementation, each of the above primary nodes (such as nodes 403, 504, 505, and 506) can correspond to one or more replica nodes / slave nodes. The replica nodes / slave nodes are replicas of the primary node and are used to store the same data as the primary node. Their main functions include data backup and load balancing. Specifically, when the primary node fails, the replica node can be promoted to the primary node to continue providing services to the client, thus ensuring service continuity. By dispersing read requests to multiple replica nodes, the pressure on the primary node can be reduced, and the overall performance and availability of the system can be improved. For a primary node, the primary node and the replica nodes corresponding to the primary node can form a service group, and the replica nodes corresponding to the primary node can perform data backup / data synchronization for the primary node within the same service group.

[0116] Figure 6 FIG. 600 is a schematic hardware structure diagram of a computing device provided by an embodiment of the present disclosure. The computing device can be a specific implementation of the above nodes or a device that hosts and installs the above clients. As Figure 6 shown, the computing device 600 includes: a bus 602, a processor 606, a memory 606, and a communication interface 608. The processor 606, the memory 606, and the communication interface 608 communicate with each other through the bus 602. The computing device 600 can be a server or a terminal device. It should be understood that the present disclosure does not limit the number of processors and memories in the computing device 600.

[0117] The bus 602 can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, Figure 6 it is represented by only one line in Figure 6 , but it does not mean that there is only one bus or one type of bus. The bus 602 can include a path for transmitting information between various components of the computing device 600 (for example, the memory 606, the processor 606, and the communication interface 608).

[0118] The processor 606 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0119] The memory 606 can include volatile memory, such as random access memory (RAM). The processor 606 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0120] The memory 606 stores executable program code, and the processor 606 executes the executable program code to implement the functions of the client and the server, thereby implementing the data processing method. That is to say, the memory 606 stores instructions for executing the data processing method.

[0121] The communication interface 608 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or a communication network.

[0122] Some steps of the data processing method provided in the embodiments of this application can be executed by a client, which can be an application installed and running on a computing device. In this case, the computing device can be a terminal. The embodiments of this application do not limit the device form of the terminal. The device for implementing the functions of the terminal can be the terminal; it can also be a device capable of supporting the terminal to implement this function, such as a chip system. This device can be installed in the terminal or used in matching with the terminal. In the embodiments of this application, the chip system can be composed of chips or can also include chips and other discrete devices.

[0123] Some steps of the data processing method provided in the embodiments of this application can be executed by a server, which can be an application installed and running on a computing device. In this case, the computing device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (content delivery network, CDN), and big data and artificial intelligence platforms.

[0124] The embodiments of this application do not limit the device form of the server. The device for implementing the functions of the server can be the server; it can also be a device capable of supporting the server to implement this function, such as a chip system. This device can be installed in the server or used in matching with the server. In the embodiments of this application, the chip system can be composed of chips or can also include chips and other discrete devices.

[0125] Next, in conjunction with Figures 7 to 17 , the data processing method provided in the embodiments of this application will be described.

[0126] In conjunction with the above Figure 4 shown data processing system, in the single-machine mode, this data processing system includes at least one client and one server / node. The data processing method provided in the embodiments of this application can be applied to an in-memory key-value database, which includes at least one partition, each partition includes at least two shards, and at least two shards included in each partition correspond to the server. As Figure 7 shown, the data processing method provided in the embodiments of this application includes S701 - S706.

[0127] S701. The client obtains the target hash value of the target key.

[0128] Among them, the target key is the key of the data to be processed, and the data to be processed is stored in the in-memory key-value database in the form of key-value pairs. The target key is the unique identifier of the data to be processed in the in-memory key-value database, and can effectively store and retrieve the data to be processed.

[0129] Optionally, the client can perform a hash operation on the target key to obtain a target hash value.

[0130] Exemplarily, the above hash algorithm can be the fifth version of the Message Digest Algorithm (MD5), the Secure Hash Algorithm (SHA), or the Cyclic Redundancy Check Algorithm (CRC).

[0131] Optionally, the target hash value can also be the hash value of the hash tag of the data to be processed, and the hash tag can be a part of the above target key.

[0132] Optionally, the hash algorithm in the embodiments of the present application can be compatible with the existing Redis algorithm, and the historical data already stored in the in-memory key-value database does not need to be migrated.

[0133] S702. Based on the target hash value, the client determines the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs.

[0134] In the embodiments of the present application, one node is one database instance, one database corresponds to one partition, and one partition includes at least two shards.

[0135] Exemplarily, as Figure 8 shown, it is a schematic diagram of partitioning and sharding in the in-memory key-value database in the single-machine mode.

[0136] Specifically, the in-memory key-value database includes one node, and the node includes at least two partitions, that is, Figure 8 DB0 and DBX (X is an integer greater than or equal to 1) shown in Figure 8 Among them, for any partition (such as DB0), the partition includes at least two shards, such as

[0137] In addition, from Figure 8As can be seen, the shards included in the same partition are uniformly numbered, and the identifiers / numbers of the shards included in different partitions can be the same. That is, the identifiers / numbers of Shard0, Shard1, Shard2, and ShardN in DB0 are different from each other, and the identifier / number of Shard0 in DB0 is the same as the identifier / number of Shard0 in DBX.

[0138] In an optional implementation manner, S702 may specifically include: the client performs a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

[0139] For example, the client can divide the target hash value by the number of shards included in the target partition to obtain a remainder. This remainder is the identifier of the target shard.

[0140] In one case, when there is one partition in the above-mentioned in-memory key-value database, this partition is the target partition to which the target key belongs, and the server can directly obtain the number of shards included in the target partition.

[0141] In another case, when there are at least two partitions in the in-memory key-value database, the client can also obtain the identifier of the target partition to which the target key belongs, and determine the target partition from the at least two partitions based on the identifier of the target partition, so as to obtain the number of shards included in the target partition. That is, the client can find the target shard in the way of partition identifier + shard identifier (i.e., DB:shard).

[0142] Optionally, the number of shards included in a partition can be configured according to requirements and is not limited by the number of threads. This number of shards can also be adjusted according to the data volume, resource volume, and service level in the node.

[0143] In this implementation manner, by performing a modulo operation on the target hash value according to the number of shards included in the target partition to which the target key belongs, the client can conveniently and quickly determine the identifier of the target shard to which the target key belongs based on a simple division operation (i.e., modulo operation), thereby improving the efficiency of data processing.

[0144] S703: The client sends a data processing request to the server corresponding to the target shard. Correspondingly, the server receives the data processing request from the client.

[0145] Among them, the data processing request includes the target hash value of the target key and the identifier of the above-mentioned target partition, and this data processing request is used to request the server to perform the operation corresponding to the data to be processed.

[0146] It should be understood that since this embodiment is a memory key-value database / data processing system in a single-machine mode, there is one service (i.e., node) in this memory key-value database / data processing system.

[0147] S704. The server determines the identifier of the target shard to which the target key belongs from at least two shards included in the target partition based on the target hash value.

[0148] In an optional implementation manner, S704 may specifically include: The server performs a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

[0149] It should be understood that the specific process of the server performing a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard is the same as or similar to the above explanation of the client performing a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard, and will not be elaborated here.

[0150] In an implementation manner of the embodiment of the present application, the shards in one partition are uniformly numbered.

[0151] In this implementation manner, since the shards in one partition are uniformly numbered, that is, the identifiers of the shards included in the same partition do not repeat. In this way, the server can determine the identifier of the target shard to which the target key belongs uniquely from at least two shards included in the target partition, and thus realize the processing process of the data to be processed based on this unique target shard, which can improve the accuracy of data processing.

[0152] Optionally, the identifier of the target shard may also be included in the above data processing request, so that the server can directly obtain the identifier of the target shard without determining the identifier of the target shard based on the target hash value. This can reduce the operation process of the server and improve the computing power of the server.

[0153] S705. The server determines the identifier of the target hash bucket where the target key is located from the hash table corresponding to the target shard based on the target hash value.

[0154] It should be understood that for each shard, one shard corresponds to one hash table, and one hash table includes one or more hash buckets.

[0155] In an optional implementation manner, the above S705 may specifically include: The server performs a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard to obtain the identifier of the target hash bucket.

[0156] Specifically, the server can divide the target hash value by the number of hash buckets included in the hash table corresponding to the target shard, so as to obtain a remainder. This remainder is the identifier of the target hash bucket.

[0157] In the embodiments of the present application, by performing a modulo operation on the target hash value according to the number of shards included in the target partition to which the target key belongs, and performing a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard, the client can conveniently and quickly determine the identifier of the target hash bucket where the target key is located based on two simple division operations (i.e., modulo operations). Thus, the operation corresponding to the data to be processed can be performed on the target hash bucket, which can improve the data processing efficiency of the data to be processed.

[0158] S706. The server performs an operation corresponding to the data to be processed on the target hash bucket.

[0159] Specifically, the server performing an operation corresponding to the data to be processed on the target hash bucket can also be understood as the server performing this operation on the data to be processed in the target hash bucket.

[0160] Among them, the operations corresponding to the data to be processed include a write operation (SET) and a read operation (GET).

[0161] In an alternative implementation, if the operation is a read operation, the above data processing request further includes a target key. At this time, the server can read the value corresponding to the target key for the target hash bucket.

[0162] In another alternative implementation, if the operation is a write operation, the above data processing request further includes a target key and a target value. At this time, the server can write the key-value pair composed of the target key and the target value into the target hash bucket.

[0163] It should be understood that when the above operation is a write operation and the value corresponding to the target key does not exist in the target hash bucket, this operation (or write operation) can also be understood as an add operation. When the operation is a write operation and the value corresponding to the target key exists in the target hash bucket, this operation (or write operation) can also be understood as an update operation. That is, by performing this operation on the data to be processed in the target hash bucket, the server can update the value originally corresponding to the target key to the above target value.

[0164] Exemplarily, such as Figure 9As shown in the figure, the data processing system includes client 901, client 902, and server 903. Among them, the server 903 includes two partitions, namely partition 903a and partition 903b. Partition 903a includes 8 shards, namely shard 903a1, shard 903a2, shard 903a3, shard 903a4, shard 903a5, shard 903a6, shard 903a7, and shard 903a8. Partition 903b also includes 8 shards, namely shard 903b1, shard 903b2, shard 903b3, shard 903b4, shard 903b5, shard 903b6, shard 903b7, and shard 903b8.

[0165] Suppose client 901 sends a first data processing request to server 903. The first data processing request includes the identifier of partition 903a and is used to indicate a write operation, specifically SET key0 value0. After receiving the first data processing request, the server 903 performs a modulo operation on the hash value of key0. Suppose the value obtained after the server 903 performs the modulo operation on the hash value of key0 is 2, indicating that the identifier of the shard to which key0 belongs is 2.

[0166] Also suppose that the identifier of shard 903a3 in partition 903a is 2. Then the server 903 can determine the hash bucket where key0 is located from the hash table corresponding to shard 903a3 and insert the key-value pair key0:value0 into this hash bucket.

[0167] In addition, it can also be supposed that client 902 sends a second data processing request to the server. The second data processing request includes the identifier of partition 903b and is used to indicate a read operation, specifically GET key0. Also suppose that the identifier of shard 903b3 in partition 903b is 2. Then the server 903 can determine the hash bucket where key0 is located from the hash table corresponding to shard 903b3 and read the value of key0 from this hash bucket.

[0168] In the embodiments of this application, after the client obtains the target hash value of the target key, it can determine the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs. Since one shard corresponds to one node / server, in this way, the client can accurately determine the server corresponding to the target shard based on the identifier of the target shard, and thus send a data processing request to this server to request this server to perform the operation corresponding to the data to be processed. It is possible to conveniently and quickly determine the server storing the data to be processed based on the identifier of the target shard to which the target key belongs, thereby improving the efficiency of data processing.

[0169] After the server receives a data processing request, since the data processing request includes the target hash value of the target key, the server does not need to perform a hash operation on the target key. Instead, based on the target hash value, it can directly determine the identifier of the target shard to which the target key belongs and the identifier of the target hash bucket where the target key is located. Then, it can perform the operations corresponding to the data to be processed on the target hash bucket, which can improve the efficiency of data processing.

[0170] Combined with the above Figure 5 In the data processing system shown in the above, in the cluster mode, the data processing system includes at least one client and at least two servers / nodes. The data processing method provided by the embodiments of the present application can be applied to an in-memory key-value database, which includes at least one partition, and each partition includes at least two shards, and each shard corresponds to a server. As Figure 10 shown, the data processing method provided by the embodiments of the present application includes S1001 - S1006.

[0171] S1001. The client obtains the target hash value of the target key.

[0172] S1002. The client determines the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value.

[0173] Exemplarily, as Figure 11 shown, it is a schematic diagram of partitioning and sharding in an in-memory key-value database in the cluster mode.

[0174] Specifically, the in-memory key-value database includes at least two nodes, namely node 0 and node Y (Y is an integer greater than or equal to 1). Each node includes at least two partitions, namely Figure 11 DB0 and DBZ (Z is an integer greater than or equal to 1) shown in the above. Among them, for any partition (such as DB0), this partition includes at least two shards, such as Figure 11 Shard0, Shard1, Shard2, ShardM - 1 (M is an integer greater than or equal to 4), ShardM, ShardM + 1, ShardM + 2, and ShardQ (Q is an integer greater than or equal to 8) shown in the above.

[0175] In addition, from Figure 11It can also be seen that the shards included in the same partition in different nodes are uniformly numbered, and the identifiers / numbers of the shards included in different partitions in the same node can be the same. That is, the identifiers / numbers of Shard0, Shard1, Shard2, and ShardN included in DB0 in Node 0 are different from the labels / identifiers of ShardM, ShardM+1, ShardM+2, and ShardQ included in DB0 in Node Y, and the identifier / number of Shard0 included in DB0 in Node 0 is the same as the identifier / number of Shard0 included in DBX in Node 0.

[0176] In this cluster mode, the client can find the target shard by means of node identifier + partition identifier + shard identifier (i.e., node:DB:shard).

[0177] S1003. The client sends a data processing request to the server corresponding to the target shard. Correspondingly, the server receives the data processing request from the client.

[0178] Combined with the description of the above embodiments, it should be understood that the data processing request includes the target hash value and the identifier of the target partition, and the data processing request is used to request the server to perform the operation corresponding to the data to be processed.

[0179] In some embodiments, before the client sends a data processing request to the server corresponding to the target shard, the data processing method provided by the embodiments of the present application may further include steps C to E.

[0180] Step C: The client determines the target shard array to which the target shard belongs based on the identifier of the target shard.

[0181] Wherein, the target shard array includes the identifiers of at least two consecutive shards.

[0182] It should be understood that the server can divide the target partition (specifically, the shards included in the target partition) based on the number of shards included in the target partition to obtain at least one shard array, and the at least one shard array includes the above target shard array.

[0183] Step D: The client determines the service group corresponding to the target shard array from the array-service relationship.

[0184] Wherein, the array-service relationship is used to indicate the shard arrays corresponding to one or more service groups.

[0185] Optionally, a service group includes a primary node and at least one replica node.

[0186] Step E: The client determines the primary node included in the service group corresponding to the target shard array as the server corresponding to the target shard.

[0187] Exemplarily, as Figure 11 shown, the data processing system includes 3 clients (i.e., client 1101, client 1102, and client 1103) and 3 service groups (i.e., service group 1104, service group 1105, and service group 1106). Among them, service group 1104 includes a primary node 1104a and a replica node 1104b, service group 1105 includes a primary node 1105a and a replica node 1105b, and service group 1106 includes a primary node 1106a and a replica node 1106b. The shard arrays corresponding to the 3 service groups are shard array A, shard array B, and shard array C respectively. Shard array A, shard array B, and shard array C are [0 - 5500], [5501 - 11000], and [11001 - 16383] respectively.

[0188] Assume that the identifier of the above target shard is 5500. Then client 1101 determines that the target shard array is shard array A. Client 1101 determines the primary node 1104a as the server corresponding to the target shard, and client 1101 sends a data processing request to the primary node 1104a.

[0189] In the embodiments of the present application, the client can determine the target shard array to which the target shard belongs and the service group corresponding to the target shard array based on the identifier of the target shard, so as to determine the primary node included in the service group as the server corresponding to the target shard. It can conveniently and quickly determine the server corresponding to the target shard based on the identifier of the target shard, which can improve the efficiency of data processing.

[0190] In an alternative implementation, the number of shards included in a partition is the maximum capacity value of the key space of the in-memory key-value database.

[0191] It should be understood that a partition can be distributed on different nodes, and the number of shards included in the partition is the number of all shards included (or corresponding) in the different nodes where the partition is distributed.

[0192] Exemplarily, in combination with the above Figure 3 example, the maximum capacity value of the key space of the in-memory key-value database is 16384. Specifically, Figure 3 the total number of hash slots corresponding to the 3 nodes in is 16384, that is, the number of shards included in a partition in the embodiments of the present application can also be 16384.

[0193] After the client 301 determines that the node corresponding to the hash slot of the key is node 302, the client 301 may send the above data processing request to node 302. Thus, after receiving the data processing request, node 302 may also implement the processing process of the data to be processed based on the data processing method provided in the embodiments of the present application.

[0194] In this implementation manner, since the number of components included in a partition is the maximum capacity value of the key space of the in-memory key-value database, in this in-memory key-value database, the key space / data space can be fully partitioned. While improving the memory usage efficiency, it can meet the multi-concurrency scenario to support the data interaction and processing process between multiple clients and multiple servers.

[0195] S1004. The server determines the identifier of the target shard to which the target key belongs from at least two shards included in the target partition based on the target hash value.

[0196] S1005. The server determines the identifier of the target hash bucket where the target key is located from the hash table corresponding to the target shard based on the target hash value.

[0197] S1006. The server executes the operation corresponding to the data to be processed for the target hash bucket.

[0198] It can be understood that the specific processes of S1001-S1006 are the same as or similar to the explanations of S701-S706 above, and will not be elaborated here.

[0199] The following uses a complete example to illustrate the data processing method provided in the embodiments of the present application.

[0200] As Figure 12 shown, the data processing system includes 1 client (i.e., client 1201) and 3 nodes (including node 1202, node 1203, and node 1204). The shard arrays corresponding to the 3 nodes are [0-5500], [5501-11000], and [11001-16383] respectively.

[0201] After the client 1201 obtains the target key, it may perform a hash operation on the target key to obtain a target hash value, corresponding to Figure 12 H = Hash(key) in. Then, the client 1201 performs a modulo operation on the target hash value according to the number of shards included in the target partition to which the target key belongs (i.e., 16384) to obtain the identifier of the target shard to which the target key belongs, corresponding to Figure 12where Shard = H % 16384. Then, the client 1201 determines the service group corresponding to the target shard array based on the array service relationship, and thus determines the primary node included in the service group corresponding to the target shard array as the node (or server) corresponding to the target shard, corresponding to Figure 12 where Node = Array[Shard].

[0202] Assume that the identifier of the target shard is 5501. Then, the client 1201 determines that the node corresponding to the target shard is node 1203. At this time, the client 1201 sends a data processing request to the node 1203.

[0203] After receiving the data processing request, the node 1203 can also perform a modulo operation on the target hash value according to the number of shards included in the target partition to which the target key belongs (i.e., 16384), and obtain the identifier of the target shard to which the target key belongs, that is, the obtained identifier of the target shard is 5501. After that, the node 1203 also determines the identifier of the target hash bucket (i.e., Figure 12 the hash bucket included in the Buckets pointed to by the arrow of shard5501 in

[0204] In some embodiments related to Valkey, the I / O thread is a read-write thread, which can receive I / O commands to be executed from the main thread. The I / O commands may include reading commands from the client, returning responses to the client, polling I / O events on a transmission control protocol (TCP) connection, and releasing memory, etc. When the I / O thread processes the I / O commands, the main thread can use more time to execute other commands.

[0205] In this embodiment, the main thread is responsible for coordinating all commands / tasks distributed to the I / O threads to ensure that no race conditions occur. The main thread can adjust the number of I / O threads participating in the execution of commands according to the current load to ensure the efficient utilization of the underlying hardware. Although the I / O threads are dynamic, the main thread will maintain thread affinity, specifically to ensure that, if possible, the I / O commands of the same client are processed by the same I / O thread to improve the locality of memory access.

[0206] In an alternative implementation, in the single-machine mode, each process in the in-memory key-value database only allows the main thread to access the database to ensure data consistency. However, when the main thread receives a large number of commands from the I / O threads, since only the main thread is allowed to access the database, the pressure on the main thread is relatively large, which will cause a bottleneck in data processing.

[0207] In some embodiments related to Dragonfly, the entire database can be divided into smaller and more independent parts, which are called database shards. Each database shard is assigned a specific thread, which can be called a shard thread, so as to achieve parallel processing of commands. This embodiment eliminates the above-mentioned single-thread bottleneck, enabling the in-memory key-value database to handle higher loads.

[0208] In this embodiment, Dragonfly divides the database into multiple database shards, and the number of database shards is less than or equal to the number of threads included in the node. To ensure data consistency, each database shard is owned and accessed by a shard thread. However, when each shard thread receives a large number of commands from I / O threads, due to only allowing the shard thread to access the database shard, the pressure on the shard thread will also be relatively large, which will also cause a bottleneck in data processing.

[0209] Based on this, the embodiments of the present application provide a data processing method, device, system, and storage medium, which are applied to database scenarios (including but not limited to data processing, data sharding, data retrieval, and data storage, etc.). After receiving a data processing request sent by a client, the server can parse and process the data processing request by controlling the target I / O thread, which can save the processing resources of the main thread in the server, avoid causing too much pressure on the main thread, and improve the availability of the threads included in the server.

[0210] Combined with Figure 7 , as Figure 13 shown, in an implementation manner of the embodiment of the present application, the above-mentioned server performs operations corresponding to the data to be processed for the target hash bucket, which may specifically include S706a - S706b.

[0211] S706a. The server parses and processes the data processing request to obtain the type of the data processing request.

[0212] In the embodiments of the present application, the types of data processing requests include I / O requests and non-I / O requests.

[0213] S706b. The server performs operations corresponding to the data to be processed for the target hash bucket based on the type of the data processing request.

[0214] It should be understood that the processing process of the data processing request includes two stages: the parsing stage and the execution stage. The server first performs the parsing stage and then the execution stage. This execution stage is the process of the server performing operations corresponding to the data to be processed.

[0215] In this implementation, since the processing priorities of different types of data processing requests are different, for example, the processing priority of an I / O request is higher than that of a non-I / O request, the server can determine the priority of a data processing request based on the type of the data processing request, and thus preferentially process the data processing request with a higher priority / importance level.

[0216] In some embodiments, the server can control the main thread to parse and process the data processing request.

[0217] In some other embodiments, the above S706b may specifically include: the server controls the target I / O thread to parse and process the data processing request to obtain the type of the data processing request.

[0218] Wherein, the target I / O thread corresponds to the above client.

[0219] It should be understood that the target I / O thread is the I / O thread in the server that receives the above data processing request.

[0220] In this embodiment, by controlling the target I / O thread to parse and process the data processing request, the server can save the processing resources of the main thread in the server, avoid causing a large pressure on the main thread, and can improve the availability of the threads included in the server.

[0221] The following describes several specific implementation manners of S706b:

[0222] Manner 1: The above S706b may specifically include S706b1 - S706b2.

[0223] S706b1. When the type of the data processing request includes an I / O request, the server controls the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0224] Wherein, the I / O request includes a read request (also referred to as a read operation) and a write request (also referred to as a write operation).

[0225] Exemplarily, as Figure 14 shown, the I / O thread can first read the data processing request through a read socket, and then process the data processing request, including a parsing stage and an execution stage. After the execution stage of the data processing request is completed, the I / O thread can also implement the write operation of the data to be processed through a write socket.

[0226] In addition, in Figure 14 the example, after the write operation of the data to be processed is executed, the I / O thread can wait for the reading process of the next data processing request to be executed.

[0227] S706b2. When the type of the data processing request includes a non-I / O request, the server controls the worker thread to perform an operation on the to-be-processed data for the target hash bucket based on the type of the data processing request.

[0228] Among them, the worker thread can be an idle worker thread in the server.

[0229] In this implementation manner, when the type of the data processing request includes an I / O request, it indicates that the priority of this data processing request is relatively high, and the server can control the target I / O thread to process this data processing request. When the type of the data processing request includes a non-I / O request, it indicates that the priority of this data processing request is relatively low, and the server can control the worker thread to process this data processing request. That is, the server can determine the priority / importance level of the data processing request based on the type of the data processing request, so as to control the target I / O thread to process requests / tasks with a higher importance level, and control the worker thread to process requests / tasks with a lower importance level, which can improve the efficiency of data processing.

[0230] Optionally, the above non-I / O requests include: KEYS, FLUSHALL or bigkey.

[0231] It should be understood that KEYS is a command used to find all keys that match a specific / given pattern, and the efficiency of this command is relatively low and may block the server. FLUSHALL is a command used to empty all databases on the server, mainly used to delete all key-value pairs on the server. And bigkey is a command used to detect and identify keys that occupy a large amount of memory. For these keys, their values are often too large or contain a large amount of data.

[0232] In this embodiment, since the non-I / O requests include KEYS, FLUSHALL or bigkey, when the type of the data processing request includes a non-I / O request, it indicates that the priority of this data processing request is relatively low (or the processing efficiency of this data processing request is relatively low). At this time, the server can allocate this data processing request to the worker thread and control the worker thread to process this data processing request. It can avoid requests / tasks with low processing efficiency from blocking the I / O thread, ensure that the I / O thread can promptly respond to requests / tasks with a higher priority / processing efficiency, and improve the stability of the data processing system.

[0233] Method 2: The above S706b can specifically include S706b3 - S706b5.

[0234] S706b3. The server estimates the processing duration for executing the data processing request based on the type of the data processing request.

[0235] Optionally, one type can correspond to a processing duration, and the server can estimate the processing duration corresponding to the type of the data processing request as the processing duration for executing the data processing request.

[0236] S706b4. When the processing duration is greater than or equal to the duration threshold, the server controls the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket.

[0237] It should be understood that when the processing duration is greater than or equal to the duration threshold, it indicates that the processing duration of the data processing request is relatively long. At this time, the server can control the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, specifically to process requests / tasks with a relatively long required duration by the worker thread.

[0238] S706b5. When the processing duration is less than the duration threshold, the server controls the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket.

[0239] It can be understood that when the processing duration is less than the duration threshold, it indicates that the processing duration of the data processing request is relatively short. At this time, the server can control the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, specifically to process requests / tasks with a relatively short required duration by the target I / O thread.

[0240] In this implementation manner, by estimating the processing duration for executing the data processing request based on the type of the data processing request, the server can control the worker thread to process requests / tasks with a relatively long required duration, and control the target I / O thread to process requests / tasks with a relatively short required duration. While improving the data processing efficiency, it can avoid the long-term occupation of the I / O thread and improve the processing performance of the I / O thread.

[0241] Method 3: The above S706b can include S706b6 - S706b8.

[0242] S706b6. The server estimates the data volume for executing the data processing request based on the type of the data processing request.

[0243] Optionally, one type can correspond to a data volume, and the server can estimate the data volume corresponding to the type of the data processing request as the data volume for executing the data processing request.

[0244] S706b7. When the data volume is greater than or equal to the preset data volume, the server controls the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket.

[0245] It should be understood that when the data volume is greater than or equal to the preset data volume, it indicates that the data volume targeted by the data processing request is large. At this time, the server can control the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, specifically, the worker thread processes the request / task with a large data volume targeted.

[0246] S706b8. When the data volume is less than the preset data volume, the server controls the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket.

[0247] It can be understood that when the data volume is less than the preset data volume, it indicates that the data volume targeted by the data processing request is small. At this time, the server can control the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, specifically, the target I / O thread processes the request / task with a small data volume targeted.

[0248] In this implementation, by estimating the data volume targeted by the data processing request based on the type of the data processing request, the server can control the worker thread to process the request / task with a large data volume targeted, and control the target I / O thread to process the request / task with a small data volume targeted. It can avoid the I / O thread from being blocked by the request / task with a large data volume, and ensure that the server (or the I / O thread) can respond to the request / task with a small data volume in a timely manner.

[0249] The following describes the process in which the server in the embodiment of the present application controls the target I / O thread and the worker thread to perform the operations corresponding to the to-be-processed data respectively in combination with the above embodiments.

[0250] As Figure 15 shown, the data processing system includes a client 1501, a client 1502, and a server 1503. Among them, the server 1503 includes an I / O thread 1, an I / O thread 2, and a worker thread 1. The I / O thread 1 corresponds to the client 1501, and the I / O thread 2 corresponds to the client 1502. The partition 1504 is a partition included in the server 1503. The partition 1504 includes 8 shards, namely shard 1504a, shard 1504b, shard 1504c, shard 1504d, shard 1504e, shard 1504f, shard 1504g, and shard 1504h.

[0251] Assume that the client 1501 sends a third data processing request to the server 1503. The third data processing request is used to indicate a read operation, specifically GET key0. Assume that the value obtained after the server 1503 performs a modulo operation on the hash value of key0 is 2, indicating that the identifier of the shard to which key0 belongs is 2.

[0252] Also assume that the identifier of shard 1504c in partition 1504 is 2. After the server 1503 receives the third data processing request, it controls the I / O thread 1 to determine the hash bucket where key0 is located from the hash table corresponding to shard 1504c, and reads the value of key0 from this hash bucket.

[0253] It can also be assumed that the client 1502 sends a fourth data processing request to the server 1503. After receiving the fourth data processing request, if the server 1503 determines that the type of the fourth data processing request includes KEYS, then the server 1503 controls the worker thread 1 to complete the processing process of the fourth data processing request.

[0254] Optionally, the above server controls the worker thread to perform the operation corresponding to the data to be processed on the target hash bucket, which can be specifically implemented through the following steps A - B.

[0255] Step A: The server adds the data processing request to the task queue corresponding to the worker thread.

[0256] Step B: The server controls the worker thread to process the data processing request in the task queue for the target hash bucket to perform the operation corresponding to the data to be processed.

[0257] It should be understood that the task queue corresponding to the worker thread can be a first-in, first-out (FIFO) queue. By adding the data processing request to the task queue corresponding to the worker thread, the server can process the tasks (including data processing requests) in the task queue in sequence when controlling the worker thread to process relevant requests / tasks, which can ensure the timing of data processing.

[0258] Exemplarily, as Figure 16 shown. The I / O thread can first read the data processing request by reading the socket, and then process the data processing request (specifically, the parsing phase in the processing process). When it is parsed that the data processing request meets the execution conditions of the above worker thread, the I / O thread can add the data processing request to the task queue corresponding to the worker thread.

[0259] After that, the worker thread can read the data processing request from this task queue and enter the execution phase. In the implementation process of the execution phase, the worker thread can implement the write operation of the data to be processed by writing to the socket.

[0260] In addition, in Figure 16In the example of , after the write operation of the data to be processed is completed, the worker thread can continue to read other tasks from the above task queue to implement the execution phase of other tasks.

[0261] In some embodiments, multiple threads in the server (including I / O threads and / or worker threads) can access the data in multiple shards included in the in-memory key-value database in parallel. If the multiple threads access a shard simultaneously, the consistency of the data in the shard can be guaranteed by a shared lock. Specifically, multiple threads can access a shard simultaneously for read operations, but only one thread can access the shard for write operations. Additionally, a thread can lock one or more shards to process multi-key commands.

[0262] Exemplarily, as Figure 17 shown, for commands that need to be processed by the I / O thread, after receiving a command from the client, the I / O thread can process the command, including the parsing phase and the execution phase. Then the I / O thread can add a shared lock to the corresponding shard and execute the command in the shard (specifically, the hash bucket included in the hash table corresponding to the shard). The execution result of the command will ultimately be returned to the client via the I / O thread.

[0263] For commands that need to be processed by the worker thread, after receiving a command from the client, the I / O thread can first perform the parsing phase on the command. After parsing and determining that the worker thread needs to be controlled to execute the command, the I / O thread assigns the command to the worker thread for processing. The worker thread performs the execution phase on the command. During the execution phase, the worker thread can also add a shared lock to the corresponding shard. Finally, the execution result of the command will be returned to the client via the worker thread.

[0264] It can be understood that in order to implement the functions in the above embodiments, the client and the server include the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combined with the units and method steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application scenario and design constraints of the technical solution.

[0265] In the above, in combination with Figures 7 to 17 , the data processing method provided according to this embodiment has been described in detail. Next, in combination with Figures 18 - 19 , the data processing device provided according to this embodiment will be described.

[0266] This application also provides a data processing device, as Figure 18As shown in the figure, the data processing device 180 is applied to an in-memory key-value database. The in-memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a hash table. The data processing device 180 includes a transceiver module 1801, a determination module 1802, and a processing module 1803.

[0267] The transceiver module 1801 is configured to receive a data processing request from a client. The data processing request includes a target hash value of a target key and an identifier of a target partition to which the target key belongs.

[0268] The determination module 1802 is configured to determine an identifier of a target shard to which the target key belongs from at least two shards included in the target partition based on the target hash value.

[0269] The determination module 1802 is further configured to determine an identifier of a target hash bucket in which the target key is located from the hash table corresponding to the target shard based on the target hash value.

[0270] The processing module 1803 is configured to perform an operation corresponding to the data to be processed on the target hash bucket.

[0271] In a possible design, the processing module 1803 is specifically configured to perform a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

[0272] The processing module 1803 is further specifically configured to perform a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard to obtain the identifier of the target hash bucket.

[0273] In a possible design, the shards in a partition are uniformly numbered.

[0274] In a possible design, the processing module 1803 is specifically configured to parse and process the data processing request to obtain the type of the data processing request.

[0275] The processing module 1803 is further specifically configured to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0276] In a possible design, the processing module 1803 is further specifically configured to control a target I / O thread to parse and process the data processing request to obtain the type of the data processing request; wherein, the target I / O thread corresponds to the client.

[0277] In a possible design, the processing module 1803 is further specifically configured to, when the type of the data processing request includes an I / O request, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0278] Alternatively, it is further specifically configured to, when the type of the data processing request includes a non-I / O request, control the worker thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

[0279] In a possible design, the above non-I / O requests include: KEYS, FLUSHALL, or bigkey.

[0280] In a possible design, the processing module 1803 is further specifically configured to estimate the processing duration for executing the data processing request based on the type of the data processing request.

[0281] The processing module 1803 is further specifically configured to, when the processing duration is greater than or equal to the duration threshold, control the worker thread to perform an operation corresponding to the data to be processed on the target hash bucket.

[0282] Alternatively, it is further specifically configured to, when the processing duration is less than the duration threshold, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket, where the target I / O thread corresponds to the client.

[0283] In a possible design, the processing module 1803 is further specifically configured to estimate the data volume for which the data processing request is directed based on the type of the data processing request.

[0284] The processing module 1803 is further specifically configured to, when the data volume is greater than or equal to the preset data volume, control the worker thread to perform an operation corresponding to the data to be processed on the target hash bucket.

[0285] Alternatively, it is further specifically configured to, when the data volume is less than the preset data volume, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket, where the target I / O thread corresponds to the client.

[0286] In a possible design, the processing module 1803 is further specifically configured to add the data processing request to the task queue corresponding to the worker thread;

[0287] The processing module 1803 is further specifically configured to control the worker thread to process the data processing request in the task queue on the target hash bucket to perform an operation corresponding to the data to be processed.

[0288] It should be understood that the above transceiver module 1801 can implement the transceiver function of the data processing device 180. For example, the transceiver module 1801 is used to execute the process of the server receiving a data processing request in S703 and S1003 of the above method embodiments. The determination module 1802 can implement the functions related to the determination process of the data processing device 180. For example, the determination module 1802 is used to execute S704, S705, S1004, and S1005 in the above method embodiments, etc. The processing module 1803 can implement the processing function of the data processing device 180. For example, the processing module 1803 is used to execute S706, S1006 in the above method embodiments and their specific implementation manners included.

[0289] The present application also provides another data processing device. As Figure 19 shown, the data processing device 190 is applied to an in-memory key-value database. The in-memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a server. The data processing device 190 includes a transceiver module 1901 and a determination module 1902.

[0290] The transceiver module 1901 is used to obtain the target hash value of the target key.

[0291] The determination module 1902 is used to determine the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value.

[0292] The transceiver module 1901 is further used to send a data processing request to the server corresponding to the target shard. The data processing request includes the target hash value and the identifier of the target partition, and the data processing request is used to request the server to execute an operation corresponding to the data to be processed.

[0293] In a possible design, the above data processing device 190 further includes a processing module 1903.

[0294] The processing module 1903 is used to perform a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

[0295] In a possible design, the determination module 1902 is further used to determine the target shard array to which the target shard belongs based on the identifier of the target shard. The target shard array includes identifiers of at least two consecutive shards.

[0296] The determination module 1902 is further used to determine the service group corresponding to the target shard array from the array service relationship based on the target shard array, where the array service relationship is used to indicate the shard arrays corresponding to one or more service groups.

[0297] The determination module 1902 is further configured to determine the primary node included in the service group corresponding to the target shard array as the server corresponding to the target shard.

[0298] It should be understood that the above transceiver module 1901 can implement the transceiver function of the data processing device 190. For example, the transceiver module 1901 is used to execute S701, S703, S1001, and S1003 in the above method embodiments. The determination module 1902 can implement the functions related to the determination process of the data processing device 190. For example, the determination module 1902 is used to execute S702, S1002, and steps C - E in the above method embodiments. The processing module 1903 can implement the processing function of the data processing device 190. For example, the processing module 1903 is used to execute the process of the client performing a modulo operation on the target hash value in the above method embodiments.

[0299] It can be understood that the technical effects of the above data processing device 180 and data processing device 190 can refer to the technical effects in the above method embodiments, and will not be elaborated here.

[0300] Among them, the transceiver module (including transceiver module 1801 and transceiver module 1901), the determination module (including determination module 1802 and determination module 1902), and the processing module (including processing module 1803 and processing module 1903) can all be implemented by software or by hardware. Exemplarily, next, taking the determination module as an example, the implementation manner of the determination module will be introduced. Similarly, the implementation manners of the transceiver module and the processing module can refer to the implementation manner of the determination module.

[0301] As an example of a software functional unit, the determination module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the determination module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running this code may be distributed in the same region, or may be distributed in different regions. Further, the multiple hosts / virtual machines / containers for running this code may be distributed in the same availability zone (AZ), or may be distributed in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Among them, generally one region may include multiple AZs.

[0302] Similarly, multiple hosts / virtual machines / containers for running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Usually, one VPC is set up within one region. For cross-region communication between two VPCs within the same region or between VPCs in different regions, a communication gateway needs to be set up within each VPC, and the interconnection between VPCs is achieved through the communication gateway.

[0303] As an example of a hardware functional unit, the determination module may include at least one computing device, such as a server. Alternatively, the determination module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above PLD may be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0304] The multiple computing devices included in the determination module can be distributed within the same region or across different regions. The multiple computing devices included in the determination module can be distributed within the same availability zone (AZ) or across different AZs. Similarly, the multiple computing devices included in the determination module can be distributed within the same VPC or across multiple VPCs. Among them, the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0305] It should be noted that in other embodiments, the determination module can be used to execute any step in the data processing method, the transceiver module can be used to execute any step in the data processing method, and the processing module can also be used to execute any step in the data processing method. The steps to be implemented by the determination module, the transceiver module, and the processing module can be specified as needed. The full functions of the data processing apparatus are realized by separately implementing different steps in the data processing method through the determination module, the transceiver module, and the processing module.

[0306] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0307] As Figure 20 shown, the computing device cluster includes at least one computing device 600. Instructions for executing a data processing method can be stored in the same manner in the memories 606 of one or more of the computing devices 600 in the computing device cluster.

[0308] In some possible implementation manners, partial instructions for executing a data processing method can also be stored separately in the memories 606 of one or more of the computing devices 600 in the computing device cluster. In other words, a combination of one or more of the computing devices 600 can jointly execute instructions for executing a data processing method.

[0309] It should be noted that the memories 606 in different computing devices 600 in the computing device cluster can store different instructions, respectively for executing partial functions of a data processing apparatus. That is to say, the instructions stored in the memories 606 of different computing devices 600 can implement the functions of one or more of a determination module, a transceiver module, and a processing module.

[0310] In some possible implementation manners, one or more of the computing devices in the computing device cluster can be connected through a network. Among them, the network can be a wide area network or a local area network, etc. Figure 21 shows a possible implementation manner. As Figure 21 shown, two computing devices 600A and 600B are connected through a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation manner, instructions for executing the functions of a transceiver module and a determination module are stored in the memory 606 of the computing device 600A. At the same time, instructions for executing the function of a processing module are stored in the memory 606 of the computing device 600B.

[0311] Figure 21 The connection manner between the computing device clusters shown can be considered in view of the fact that the data processing method provided in the present application requires a large amount of data processing procedures. Therefore, it is considered to hand over the function implemented by the processing module to the computing device 600B for execution.

[0312] It should be understood that Figure 21 the function of the computing device 600A shown can also be completed by multiple computing devices 600. Similarly, the function of the computing device 600B can also be completed by multiple computing devices 600.

[0313] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to Figure 20 and Figure 21 the connection mode of the computing device cluster. The difference is that the same instructions for executing the data processing method can be stored in the memory 606 of one or more computing devices 600 in the computing device cluster.

[0314] In some possible implementation manners, partial instructions for executing the data processing method can also be separately stored in the memory 606 of one or more computing devices 600 in the computing device cluster. In other words, the combination of one or more computing devices 600 can jointly execute the instructions for executing the data processing method.

[0315] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, at least one computing device is enabled to execute the data processing method.

[0316] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc. The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0317] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, Applied to a memory key-value database, the memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a hash table. The method includes: Receiving a data processing request from a client, the data processing request includes a target hash value of a target key and an identifier of a target partition to which the target key belongs; Based on the target hash value, determining an identifier of a target shard to which the target key belongs from at least two shards included in the target partition; Based on the target hash value, determining an identifier of a target hash bucket where the target key is located from the hash table corresponding to the target shard; Performing an operation corresponding to the data to be processed on the target hash bucket.

2. The method according to claim 1, wherein Based on the target hash value, determining an identifier of a target shard to which the target key belongs from at least two shards included in the target partition, includes: Performing a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard; Based on the target hash value, determining an identifier of a target hash bucket where the target key is located from the hash table corresponding to the target shard, includes: Performing a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard to obtain the identifier of the target hash bucket.

3. The method according to claim 1 or 2, characterized in that, The shards in a partition are uniformly numbered.

4. The method according to any one of claims 1-3, characterized in that, The performing an operation corresponding to the data to be processed on the target hash bucket includes: Parsing and processing the data processing request to obtain the type of the data processing request; Performing an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

5. The method according to claim 4, wherein The parsing and processing the data processing request to obtain the type of the data processing request includes: Controlling a target input / output (I / O) thread to parse and process the data processing request to obtain the type of the data processing request; wherein, the target I / O thread corresponds to the client.

6. The method according to claim 4 or 5, characterized in that The performing an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request includes: When the type of the data processing request includes an I / O request, controlling the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request; Or, when the type of the data processing request includes a non-I / O request, controlling a worker thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

7. The method according to claim 6, characterized in that, The non-I / O requests include: KEYS, FLUSHALL or bigkey.

8. The method according to claim 4 or 5, characterized in that, The performing an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request includes: Estimating a processing duration for executing the data processing request based on the type of the data processing request; When the processing duration is greater than or equal to a duration threshold, controlling the worker thread to perform an operation corresponding to the data to be processed on the target hash bucket; Alternatively, when the processing duration is less than the duration threshold, control the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, where the target I / O thread corresponds to the client.

9. The method according to claim 4 or 5, characterized in that Performing the operation corresponding to the to-be-processed data on the target hash bucket based on the type of the data processing request includes: Estimate the amount of data for which the data processing request is to be performed based on the type of the data processing request; When the amount of data is greater than or equal to the preset amount of data, control the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket; Alternatively, when the amount of data is less than the preset amount of data, control the target I / O thread to perform the operation corresponding to the to-be-processed data on the target hash bucket, where the target I / O thread corresponds to the client.

10. The method according to claim 8 or 9, characterized in that, Controlling the worker thread to perform the operation corresponding to the to-be-processed data on the target hash bucket includes: Add the data processing request to the task queue corresponding to the worker thread; Control the worker thread to process the data processing request in the task queue for the target hash bucket to perform the operation corresponding to the to-be-processed data.

11. A data processing method, characterized in that, Applied to a memory key-value database, the memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a server. The method includes: Obtain the target hash value of the target key; Based on the target hash value, determine the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs; Send a data processing request to the server corresponding to the target shard, the data processing request includes the target hash value and the identifier of the target partition, and the data processing request is used to request the server to perform the operation corresponding to the to-be-processed data.

12. The method according to claim 11, wherein The determining the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value includes: Perform a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard.

13. The method according to claim 11 or 12, characterized in that Before sending the data processing request to the server corresponding to the target shard, the method further includes: Determine the target shard array to which the target shard belongs based on the identifier of the target shard, where the target shard array includes identifiers of at least two consecutive shards; Determine the service group corresponding to the target shard array from the array-service relationship, where the array-service relationship is used to indicate the shard arrays corresponding to one or more service groups; Determine the primary node included in the service group corresponding to the target shard array as the server corresponding to the target shard.

14. A data processing device, characterized in that, Applied to a memory key-value database, the memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a hash table. The device includes: a transceiver module, a determination module, and a processing module; The transceiver module is configured to receive a data processing request from a client, where the data processing request includes a target hash value of a target key and an identifier of a target partition to which the target key belongs; The determination module is configured to determine, based on the target hash value, an identifier of a target shard to which the target key belongs from at least two shards included in the target partition, and determine, based on the target hash value, an identifier of a target hash bucket where the target key is located from a hash table corresponding to the target shard; The processing module is configured to perform an operation corresponding to the data to be processed on the target hash bucket.

15. The apparatus according to claim 14, wherein The processing module is specifically configured to perform a modulo operation on the target hash value according to the number of shards included in the target partition to obtain the identifier of the target shard; The processing module is further specifically configured to perform a modulo operation on the target hash value according to the number of hash buckets included in the hash table corresponding to the target shard to obtain the identifier of the target hash bucket.

16. The device according to claim 14 or 15, characterized in that The processing module is specifically configured to: Parse and process the data processing request to obtain the type of the data processing request; and perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request.

17. The apparatus according to claim 16, wherein The processing module is further configured to control a target input / output (I / O) thread to parse and process the data processing request to obtain the type of the data processing request; wherein the target I / O thread corresponds to the client.

18. The device according to claim 16 or 17, characterized in that, The processing module is specifically configured to: When the type of the data processing request includes an I / O request, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request; When the type of the data processing request includes a non-I / O request, control a worker thread to perform an operation corresponding to the data to be processed on the target hash bucket based on the type of the data processing request; Or, Estimate a processing duration for executing the data processing request based on the type of the data processing request; When the processing duration is greater than or equal to a duration threshold, control the worker thread to perform an operation corresponding to the data to be processed on the target hash bucket; when the processing duration is less than the duration threshold, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket, where the target I / O thread corresponds to the client; Or, Estimate a data volume for which the data processing request is directed based on the type of the data processing request; When the data volume is greater than or equal to a preset data volume, control the worker thread to perform an operation corresponding to the data to be processed on the target hash bucket; when the data volume is less than the preset data volume, control the target I / O thread to perform an operation corresponding to the data to be processed on the target hash bucket, where the target I / O thread corresponds to the client.

19. The apparatus according to claim 18, wherein The processing module is further configured to add the data processing request to the task queue corresponding to the worker thread; and control the worker thread to process the data processing request in the task queue for the target hash bucket to perform the operation corresponding to the data to be processed.

20. A data processing device, characterized in that, Applied to a memory key-value database, the memory key-value database includes at least one partition, each partition includes at least two shards, and each shard corresponds to a server. The apparatus includes: a transceiver module and a determination module; The transceiver module is configured to obtain the target hash value of the target key; The determination module is configured to determine the identifier of the target shard to which the target key belongs from at least two shards included in the target partition to which the target key belongs based on the target hash value; The transceiver module is further configured to send a data processing request to the server corresponding to the target shard. The data processing request includes the target hash value and the identifier of the target partition, and the data processing request is used to request the server to perform the operation corresponding to the data to be processed.

21. The device according to claim 20, wherein The determination module is further configured to: Determine the target shard array to which the target shard belongs based on the identifier of the target shard. The target shard array includes the identifiers of at least two consecutive shards; Determine the service group corresponding to the target shard array from the array-service relationship based on the target shard array, where the array-service relationship is used to indicate the shard arrays corresponding to one or more service groups; Determine the primary node included in the service group corresponding to the target shard array as the server corresponding to the target shard.

22. A data processing system, characterized in that, The data processing system includes a client and a server. The server is configured to execute the data processing method according to any one of claims 1-10, and the client is configured to execute the data processing method according to any one of claims 11-13.

23. A cluster of computing devices, characterized in that, Including at least one computing device, each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute the instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method according to any one of claims 1-10, or executes the data processing method according to any one of claims 11-13.

24. A computer program product comprising instructions, characterized in that, When the instructions are run by the computing device cluster, the computing device cluster is caused to execute the data processing method according to any one of claims 1-10, or execute the data processing method according to any one of claims 11-13.

25. A computer-readable storage medium, characterized in that, Including computer program instructions, when the computer program instructions are executed by the computing device cluster, the computing device cluster executes the data processing method according to any one of claims 1-10, or executes the data processing method according to any one of claims 11-13.

Images