Data processing method, data access method, device and computer equipment

Abstract

A data processing method, a data access method, an apparatus, and a computer device, the data processing method comprising: reading the number of accesses to each data stored in a key-value storage system; wherein, in each hash table of the key-value storage system, an ordered linked list is used to store data, and the access heat of the data in each linked list node of the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; data is stored in each balanced tree based on the key value size of the data; when the number of accesses to the target data does not belong to the access heat interval corresponding to the balanced tree where the target data is located, searching for the target access heat interval corresponding to the number of accesses to the target data; and migrating the target data to the balanced tree of the linked list node corresponding to the target access heat interval. The above method does not need to traverse all data when accessing data, which can improve the reading and writing performance of the data; and the higher the access heat, the less search time is required when accessing data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment and storage medium, and a data access method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of technology, more and more data needs to be stored. In large-scale applications with frequent interactions (such as large-scale games), data reading and writing are even more frequent. The read and write efficiency and latency of data will directly affect the user experience. In order to improve the read and write performance of data, a key-value storage system is usually deployed in the front end of the database to withstand peak data access and protect the database from crashing. Therefore, the key-value storage system in this type of application needs to have high performance and high availability.

[0003] The key-value storage engine in the prior art usually uses a hash table as the underlying data structure: first, a hash algorithm is used to calculate the hash value hash(key) of the data based on the key value (key) of the data, and a head pointer Head(n) = hash(key) of the hash table is located. According to the head pointer, all the data in the corresponding conflict list are traversed, and the target data is found by comparing the key values. If the target data is not in the conflict chain, the data can be inserted at the head of the conflict chain. However, in such a key-value storage method, accessing the data at the end of the hash conflict list requires traversing all the linked list nodes. The longer the data conflict list, the more time it takes and the lower the data read and write performance. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, data access method, device and computer equipment with high read and write performance to address the above technical problems.

[0005] A data processing method, comprising:

[0006] Read the access count of each data stored in the key-value storage system; wherein, in each hash table of the key-value storage system, an ordered linked list is used to store data, and the access popularity of data in each linked list node of the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; and data is stored in each balanced tree based on the key value size of the data;

[0007] When the number of accesses to the target data does not belong to the access heat interval corresponding to the balanced tree, searching for the target access heat interval corresponding to the number of accesses to the target data;

[0008] The target data is migrated to a balanced tree of linked list nodes corresponding to the target access heat interval.

[0009] In one embodiment, adjusting the first preset number or the second preset number based on the access time includes: when the access time of the data is greater than a preset access time threshold, reducing the first preset number or the second preset number.

[0010] In another embodiment, adjusting the first preset number or the second preset number based on the migration number of each data includes: if the migration number is greater than a preset migration number threshold, increasing the first preset number or the second preset number.

[0011] In one embodiment, the key value corresponding to the left node of the balanced tree is smaller than the key value corresponding to the parent node, and the key value corresponding to the parent node is smaller than the key value corresponding to the right node; and sequentially traversing the balanced tree in each linked list node of the target hash table to search for the key value corresponding to the data access request includes:

[0012] A current linked list node is selected in the ordered linked list of the target hash table, and the balanced tree of the current linked list node is traversed starting from the root node; in the balanced tree: the key value of the data access request is compared with the key value of the root node, if the key value of the data access request is less than the key value of the root node, the left node of the root node is accessed; if the key value of the data access request is greater than the key value of the root node, the right node of the root node is accessed; if the key value of the data access request is not found in the balanced tree of the current linked list node, the balanced tree of the next linked list node is jumped to for searching.

[0013] A data access method, the method comprising: sending a data access request to a server, the data access request being used to enable the server to determine a corresponding hash value based on a key value of the data access request; searching a corresponding target hash table in a key-value storage system based on the hash value; sequentially traversing a balanced tree in each linked list node of the target hash table to search for a key value corresponding to the data access request; and when the key value corresponding to the data access request is found in the balanced tree, reading data to be accessed stored in the node where the key value is located;

[0014] Receive the data to be accessed fed back by the server.

[0015] A data processing device, comprising:

[0016] An access count reading module is configured to read the access count of each data item stored in the key-value storage system. Each hash table in the key-value storage system uses an ordered linked list to store data, with the access popularity of the data in each linked list node decreasing in sequence. A balanced tree is stored in each linked list node of the ordered linked list. Each balanced tree stores data based on the size of the data's key value.

[0017] The access count detection module is used to find the target access heat interval corresponding to the access count of the target data when the access count of the target data does not belong to the access heat interval corresponding to the balanced tree;

[0018] A data migration module is used to migrate the target data to a balanced tree of linked list nodes corresponding to the target access heat interval.

[0019] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned data processing method and data access method when executing the computer program.

[0020] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned data processing method and data access method when executed by a processor.

[0021] In the above-mentioned data processing method, data access method, device and computer equipment, in each hash table of the key-value storage system, an ordered linked list is used to store data based on data access heat, and the data access heat of each linked list node in the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; for each data stored in the key-value storage system, data migration is judged based on the number of accesses of each data, if the number of accesses of the target data does not belong to the access heat interval corresponding to the balanced tree, the target access heat interval corresponding to the number of accesses of the target data is searched, and the target data is migrated to the balanced tree of the linked list node corresponding to the target access heat interval. In the above method, in the key-value storage system, an ordered linked list is used to store data based on the data access popularity, and at the same time, multiple data are stored in each linked list node of the ordered linked list in the form of a balanced tree, and the data is stored based on the key value size in the balanced tree. Therefore, when accessing data, the key value corresponding to the data access request can be found in each balanced tree of the ordered linked list based on the comparison between the key value and the key value size of each node. There is no need to traverse all the data in the hash table, the time required is short, and the reading and writing performance of the data can be improved; and the higher the access popularity of the data, the less search time is required when accessing it. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A diagram of an application environment of a data processing method in one embodiment;

[0023] Figure 2 1 is a flow chart of a data processing method in one embodiment;

[0024] Figure 3 is a flow chart of a data processing method in another embodiment;

[0025] Figure 4is a flow chart of a data processing method in another embodiment;

[0026] Figure 5 A storage diagram of a key-value storage system in a specific embodiment

[0027] Figure 6 is a schematic diagram of data migration in a specific embodiment;

[0028] Figure 7 Schematic diagram of adaptive adjustment of the M value in a specific embodiment;

[0029] Figure 8 is a structural block diagram of a data processing device in one embodiment;

[0030] Figure 9 is a structural block diagram of a data processing device in another embodiment;

[0031] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0033] The data processing method provided in this application can be applied to Figure 1In the application environment shown. The terminal 102 communicates with the server 104 through the network. In each hash table of the key-value storage system of the server 104, an ordered linked list is used to store data based on the data access heat, and the data access heat in each linked list node of the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; the server 104 judges the data migration based on the number of accesses of each data stored in the key-value storage system. If the number of accesses of the target data does not belong to the access heat interval corresponding to the balanced tree, the target access heat interval corresponding to the number of accesses of the target data is searched, and the target data is migrated to the balanced tree of the linked list node corresponding to the target access heat interval. The terminal 102 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. Server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0034] In one embodiment, the data storage in the data processing method provided herein can be implemented through cloud storage. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in a network through application software or application interfaces to work together and provide external data storage and service access functions.

[0035] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0036] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0037] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 The server in is used as an example to illustrate, including steps S210 to S230.

[0038] Step S210: Read the access count of each data stored in the key-value storage system.

[0039] Among them, in each hash table of the key-value storage system, an ordered linked list is used to store data, and the data access popularity in each linked list node of the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; each balanced tree stores data based on the key value size of the data.

[0040] Among them, the key-value storage system stores data in the form of key-value pairs; the key-value distributed storage system has a fast query speed, stores a large amount of data, supports high concurrency, and is very suitable for querying through primary keys; in this embodiment, the key-value storage system uses a hash table as the underlying data structure; the key-value storage system includes multiple hash tables (Hashtable, which is a data structure that is directly accessed according to the key value), and each hash table contains an ordered linked list (starting from the head node to the end of the linked list, the data in the nodes are arranged in order); in this embodiment, the data in the ordered linked list is sorted based on the access popularity of each data; further, the data access popularity in each linked list node of the ordered linked list decreases in sequence; that is, in the first linked list node of the ordered linked list, the data access popularity is the highest, that is, the head pointer of each hash table points to the linked list node corresponding to the data with the highest access popularity in the hash table.

[0041] Furthermore, each node in the ordered linked list stores a balanced tree, and the data stored in each node of each balanced tree is based on the size of the data's key value. It can be understood that the head pointer of the hash table points to the balanced tree corresponding to the most popular data. A balanced tree is one in which the height difference of the subtrees at any node is less than or equal to 1. Common balanced trees include B-trees (multi-way balanced search trees) and AVL trees (binary balanced search trees).

[0042] Furthermore, in one embodiment, the ordered linked lists in each hash table include ordered circular linked lists. A circular linked list can be formed by connecting the ends of the linked lists. Using ordered circular linked lists for data storage in each hash table of the key-value storage system allows traversal from any node in the linked list to all nodes in the linked list, which can improve search efficiency when accessing data in some cases.

[0043] In one embodiment, the key values ​​of the left node, parent node, and right node of a balanced tree are arranged in order of size. In a specific embodiment, the key value corresponding to the left node of the balanced tree is smaller than the key value corresponding to the parent node, and the key value corresponding to the parent node is smaller than the key value corresponding to the right node. It is understood that in this embodiment, the storage location of data in the balanced tree is determined by the size of the key value. When accessing data, the key value of the data to be accessed can be compared with the key value of each node to find the node where the data to be accessed is located. For example, the key value of the data to be accessed is compared with the key value of the root node of the balanced tree. If it is smaller than the key value of the root node, the left node is accessed; if it is larger than the key value of the root node, the right node is accessed, and so on. If the data to be accessed is not found in the current balanced tree, the search is performed in the balanced tree of the next linked list node in the ordered linked list. Because a balanced tree is used and data is stored based on the data's key value, querying each balanced tree when accessing data is more efficient and less time-consuming than a traversal method. It is understood that in other embodiments, data can also be stored in the balanced tree based on the data's key value in other ways, such as the left node's key value being larger than the right node's key value. In one embodiment, assuming the number of nodes in the balanced tree is N, the height of the balanced tree is log2(N).

[0044] In this embodiment, after the data is stored in the key-value access system, the number of accesses is recorded for each access to the data. In this embodiment, the number of accesses to each piece of data is read and detected.

[0045] In one embodiment, when the data migration detection condition is met, the step of reading the access count of each data stored in the key-value storage system is entered.

[0046] The data migration detection condition is used to limit the conditions under which the server starts to read and detect the access counts of each data. Only when it is detected that the data migration detection condition is met, the access counts of each data stored in the key-value storage system are read and detected.

[0047] Furthermore, in one embodiment, when the number of accesses to all data in the key-value storage system reaches a first preset number, it is determined that the data migration detection condition is met; or in another embodiment, when the number of accesses to the same data in the key-value storage system within a preset time period reaches a second preset number, it is determined that the data migration detection condition is met.

[0048] The number of times all data stored in the key-value storage system are accessed is counted and recorded. When the total number of times all data are accessed reaches a first preset number, it is determined that the data migration detection condition is met. As the number of accesses increases, the access popularity corresponding to each data stored in the key-value storage system is likely to change. Therefore, in this embodiment, when the total number of times all data stored in the key-value storage system are accessed reaches a certain value, it is determined that the data migration detection condition is met, and at this time, the step of reading the number of times each data stored in the key-value storage system is accessed is started.

[0049] In another embodiment, the number of accesses to each data item in the key-value storage system is individually counted and monitored. If the number of accesses to a particular data item reaches a first predetermined number, this indicates that the access popularity of the data item has changed, and its storage location in the key-value storage system needs to be adjusted. Furthermore, in other embodiments, the data migration detection condition may be determined to be satisfied when the number of data items with a second predetermined number of accesses reaches a predetermined number.

[0050] In the above embodiment, by setting data migration detection conditions to trigger the detection and judgment of the number of accesses to each data stored in the key-value storage system, frequent changes in access frequency can be avoided, resulting in frequent migration of data nodes and thus affecting CPU (central processing unit) performance; wherein, the first preset number and the second preset number can be set according to actual conditions.

[0051] Step S220 : When the number of accesses to the target data does not belong to the access heat interval corresponding to the corresponding balanced tree, searching for the target access heat interval corresponding to the number of accesses to the target data.

[0052] Since the data in the ordered linked list in any hash table is stored after being sorted based on the access popularity of the data, the balanced tree in each linked list node corresponds to an access popularity interval. When it is detected that the number of accesses to the data stored in this balanced tree has changed and no longer belongs to the access popularity interval corresponding to the balanced tree, the data is migrated. First, the new access popularity is determined based on the number of accesses to the data, thereby determining the access popularity interval currently corresponding to the data, which is recorded as the target access popularity interval in this embodiment.

[0053] In one embodiment, the access popularity of data represents the number of times the data is accessed, or the access popularity of data represents the number of times the data is accessed within a predetermined time period; it is understandable that in other embodiments, the access popularity of data can also be defined in other ways.

[0054] Step S230: Migrate the target data to the balanced tree of the linked list node corresponding to the target access heat interval.

[0055] In step S220, the target access heat interval actually corresponding to the data to be migrated has been determined, and then the linked list node of the ordered linked list corresponding to the target access heat interval, that is, the corresponding balanced tree, can be determined; further, in the balanced tree corresponding to the target access heat, based on the key value of the data to be migrated and the key value of the data stored in each node in the balanced tree corresponding to the target access heat, the location where the data to be migrated needs to be stored in the balanced tree to which it is migrated is determined.

[0056] In one embodiment, if the access popularity of data increases, the data is migrated to the balanced tree corresponding to the target access popularity interval with more access times. If the access popularity of data decreases, the data is migrated to the balanced tree corresponding to the target access popularity interval with fewer access times for storage.

[0057] Since the data in the ordered linked lists corresponding to the hash tables in the key-value storage system in the above embodiment are stored based on access popularity, the data are searched in sequence in the ordered linked lists during access. For data with higher access popularity, the search time is shorter, thereby further improving data access performance.

[0058] In the above-mentioned data processing method, in each hash table of the key-value storage system, an ordered linked list is used to store data based on the data access heat, and the data access heat of each linked list node in the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; for each data stored in the key-value storage system, data migration is judged based on the number of accesses, and if the number of accesses of the target data does not belong to the access heat interval corresponding to the balanced tree, the target access heat interval corresponding to the number of accesses of the target data is searched, and the target data is migrated to the balanced tree of the linked list node corresponding to the target access heat interval. In the above method, in the key-value storage system, an ordered linked list is used to store data based on the data access popularity, and at the same time, multiple data are stored in each linked list node of the ordered linked list in the form of a balanced tree, and the data is stored based on the key value size in the balanced tree. Therefore, when accessing data, the key value corresponding to the data access request can be found in each balanced tree of the ordered linked list based on the comparison between the key value and the key value size of each node. There is no need to traverse all the data in the hash table, the time required is short, and the reading and writing performance of the data can be improved; and the higher the access popularity of the data, the less search time is required when accessing it.

[0059] Furthermore, in one embodiment, the method further includes: obtaining access time of each data in the key-value storage system; and adjusting the first preset number and the second preset number based on the access time.

[0060] When accessing data in a key-value storage system, not only the number of times the data is accessed is counted, but also the access time of the data is counted. If the access time of the data exceeds the preset access time threshold, it means that the setting of the first preset number of times or the second preset number of times is unreasonable. In this embodiment, the first preset number of times or the second preset number of times is adjusted based on the access time of the data.

[0061] Furthermore, in one embodiment, if the access time of the data exceeds a preset access time threshold, the first preset number of times or the second preset number of times is adjusted down. If, during the adjustment process, it is detected that the access time is lower than the preset access time threshold, the adjustment of the first preset number of times or the second preset number of times is stopped. Furthermore, if, after adjusting the first preset number of times or the second preset number of times once, the access time of the data still exceeds the preset access time threshold, the first preset number of times or the second preset number of times is continued to be lowered until it is detected that the access time of the data is less than or equal to the preset access time threshold, at which point the adjustment of the first preset number of times or the second preset number of times is stopped.

[0062] In another embodiment, the number of migrations of each data in the key-value storage system is counted; and the first preset number or the second preset number is adjusted based on the number of migrations of each data. Further, in one embodiment, if the number of migrations is greater than a preset migration number threshold, the first preset number or the second preset number is increased. In this embodiment, the number of migrations of data in the database is counted. If the number of migrations of data within a set time period exceeds the preset number threshold, it indicates that the data migration is too frequent, and the first preset number or the second preset number can be increased to reduce the number of data migrations. In one embodiment, whether the number of data migrations of each data exceeds the preset migration number threshold can be monitored within a set time period. Wherein, the preset number threshold and the set time period can be set according to actual conditions.

[0063] In one embodiment, Figure 3 As shown, the above method further includes steps S310 to S330.

[0064] Step S310: Obtain the data to be stored, and perform a hash operation on the key value of the data to be stored to obtain a corresponding hash value.

[0065] A hash algorithm transforms an input of arbitrary length into an output of fixed length, known as a hash value. This conversion is a compression mapping; that is, the space of hash values ​​is typically much smaller than the space of inputs. Different inputs may hash to the same output, making it impossible to determine a unique input value from the hash value. Simply put, it's a function that compresses a message of arbitrary length into a message digest of a fixed length.

[0066] In one embodiment, the data to be stored may be any type of data, including a key value of the data to be stored; in a specific embodiment, the data to be stored is the user's role data, and the key value of the data to be stored may be the user's ID (Identity document).

[0067] Step S320: Locate the hash table header pointer in the key-value storage system according to the hash value.

[0068] The head pointer is used to determine the storage location corresponding to the first element in the linear list. In this embodiment, the hash table uses an ordered linked list structure for data storage. The head pointer indicates the storage location corresponding to the first element in the first linked list node in the ordered linked list. In this embodiment, the hash value obtained by performing a hash operation on the key value of the data to be stored can be used to find the hash table head pointer in the key-value storage system. The data to be stored can then be stored in the corresponding hash table based on the hash table head pointer.

[0069] Step S330 , storing the data to be stored in a target balanced tree in the hash table corresponding to the hash table header pointer based on the key value of the data to be stored; wherein the data access heat corresponding to the target balanced tree is the lowest.

[0070] Since the ordered linked list of the hash table is sorted based on data access popularity, the access count of the data to be stored can be recorded as 0 when it is stored. At this time, the data to be stored is stored in the balanced tree with the lowest data access popularity in the ordered linked list, which is recorded as the target balanced tree in this embodiment. Furthermore, the position of the data to be stored in the balanced tree can be determined by comparing the key value of the data to be stored with the corresponding key values ​​of each node in the balanced tree with the lowest data access popularity. In one embodiment, the data with the lowest data access popularity means that the data has the lowest data access popularity within the same hash table.

[0071] In one embodiment, after reading the access counts of each data stored in the key-value storage system, the above method further includes: dividing the access counts of each data into regions to obtain division results; and filling each data into each balanced tree of the ordered linked list in sequence based on the division results.

[0072] In one embodiment, all data is partitioned based on the number of accesses at a predetermined time interval, and based on the partitioning results, all data is rearranged and populated into the balanced trees of the ordered linked list. In this embodiment, at regular intervals, all data is re-partitioned based on the number of accesses, and the data is re-stored into the balanced trees based on the partitioning results.

[0073] In one embodiment, Figure 4 As shown, the above data processing method further includes steps S410 to S430.

[0074] Step S410: receiving a data access request, and determining a corresponding hash value according to a key value of the data access request.

[0075] The data access request is a request for accessing data sent by the terminal, and the data access request carries a key value corresponding to the data requested for access. In one embodiment, a hash operation is performed on the key value of the data access request to obtain a hash value.

[0076] Step S420: Search the corresponding target hash table in the key-value storage system according to the hash value.

[0077] The corresponding hash table header pointer can be located according to the hash value corresponding to the key value of the data access request, and then the hash table storing the accessed data corresponding to the data access request is determined, which is recorded as the target hash table in this embodiment.

[0078] Step S430 , sequentially traverse the balanced tree in each linked list node of the target hash table to search for the key value corresponding to the data access request. When the key value corresponding to the data access request is found in the balanced tree, read the data stored in the node.

[0079] Among them, the key-value storage system stores data in the form of key-value pairs; the key-value distributed storage system has a fast query speed, stores a large amount of data, supports high concurrency, and is very suitable for querying through primary keys; in this embodiment, the key-value storage system uses a hash table as the underlying data structure; the key-value storage system includes multiple hash tables (Hashtable, which is a data structure that is directly accessed based on the key value), and each hash table contains an ordered linked list (starting from the head node to the end of the linked list, the data in the nodes are arranged in order); in this embodiment, the data in the ordered linked list is sorted based on the access popularity of each data; further, each node of the ordered linked list stores a balanced tree (the height difference of the subtree of any node is less than or equal to 1), and the data stored in each node in each balanced tree is stored based on the key value size of the data.

[0080] Furthermore, in one embodiment, the ordered linked lists in each hash table include ordered circular linked lists. Using ordered circular linked lists for data storage in each hash table of the key-value storage system allows traversal from any node in the linked list to all nodes in the linked list, thereby improving data access efficiency.

[0081] In one embodiment, the key values ​​corresponding to the left node, parent node, and right node of the balanced tree are arranged in order of size. In this embodiment, the balanced tree in each linked list node of the target hash table is traversed in sequence to find the key value corresponding to the data access request, including: selecting the current linked list node in the ordered linked list of the target hash table; traversing the balanced tree of the current linked list node starting from the root node; in the balanced tree: if the key value of the access request is less than the key value of the parent node, accessing the subtree with the key value less than the key value of the parent node; if the key value of the access request is greater than the key value of the parent node, accessing the subtree with the key value greater than the key value of the parent node; until a node with the same key value as the data access request is found; if the key value of the data access request is not found in the balanced tree of the current linked list node, jumping to the balanced tree of the next linked list node for search.

[0082] In this embodiment, the key values ​​of the left node, parent node and right node of the balanced tree are arranged in order. It can be understood that they can be arranged in ascending order, that is, the left node key value is smaller than the parent node key value, and the parent node key value is smaller than the right node key value; they can also be arranged in descending order, that is, the left node key value is greater than the parent node key value, and the parent node key value is greater than the right node key value.

[0083] Furthermore, in one specific embodiment, the key value corresponding to the left child node of the balanced tree is less than the key value corresponding to the parent node, and the key value corresponding to the parent node is less than the key value corresponding to the right node. In this embodiment, when accessing data, the key value of the data to be accessed is compared with the key value of the root node of the balanced tree. If it is less than the key value of the root node, the left node of the root node is accessed; if it is greater than the key value of the root node, the right node is accessed; and so on. If the data to be accessed is not found in the current balanced tree, the search is performed in the balanced tree of the next linked list node in the ordered linked list. In another specific embodiment, the key value corresponding to the left node of the balanced tree is greater than the key value corresponding to the parent node, and the key value corresponding to the parent node is greater than the key value corresponding to the right node; in this embodiment, when accessing data, the key value of the data to be accessed is compared with the key value of the root node of the balanced tree. If it is less than the key value of the root node, the right node of the root node is accessed; if it is greater than the key value of the root node, the left node is accessed; and so on. If the data to be accessed is not found in the current balanced tree, the search is performed in the balanced tree of the next linked list node in the ordered linked list.

[0084] In this embodiment, the key value of the data to be accessed corresponding to the data access request is compared with the key value of the data stored in each balanced tree in the target hash table in turn. Since the key value of the right node in the balanced tree is greater than the key value of the parent node, and the key value of the parent node is greater than the key value of the left node, the data to be accessed can be found based on this characteristic. First, the data to be accessed is compared with the key value of the root node of the balanced tree. If the key value is less than the key value of the root node, only the left subtree of the root node is searched. If the key value is greater than the root node, only the nodes in the right subtree of the root node are searched. Then, the key values ​​are compared with the current node as the parent node until the same key value is found, that is, the search is completed. Further, when the same key value is not found in the current balanced tree, the balanced tree of the next linked list node can be entered for search. Since a balanced tree is used, there is no need to access all the data stored in the balanced tree, and the search efficiency is high. When accessing data, the query efficiency in each balanced tree is high, which is less time-consuming than the traditional traversal method, thereby improving data access performance. It is understandable that in other embodiments, the balanced tree may also store data in other ways based on the key value of the data, for example, the left node key value is greater than the right node key value, etc. Accordingly, the search strategy is adjusted accordingly when accessing data.

[0085] In another embodiment, the present application also provides a data access method, which is applied to Figure 1 Taking the terminal in as an example, the method includes the following steps: sending a data access request to the server, wherein the data access request is used to enable the server to determine the corresponding hash value according to the key value of the data access request; searching the corresponding target hash table in the key value storage system according to the hash value; traversing the balanced tree in each linked list node of the target hash table in turn to search for the key value corresponding to the data access request, and when the key value corresponding to the data access request is found in the balanced tree, reading the data to be accessed stored in the node; and receiving the data to be accessed fed back by the server.

[0086] In this embodiment, the application scenario of data access is described from the perspective of the terminal. In this embodiment, the terminal sends a data access request to the server, and the server searches for data in the key-value storage system based on the data key value in the data access request sent by the terminal. For the specific search process, please refer to the description of the embodiment of searching for data when the server receives a data access request in the data processing method, which will not be repeated here.

[0087] This application also provides an application scenario, which applies the above-mentioned data processing method. Specifically, the application of the data processing method in this application scenario is as follows:

[0088] In this embodiment, the above data processing method is applied to a game application.

[0089] Count the number of accesses to all data in the key-value storage system and sort them, and divide the sorted data into N intervals.

[0090] like Figure 5 The figure shows a storage diagram of a key-value storage system in a specific embodiment. Each interval of data is sequentially inserted into N balanced trees. Each parent node in the balanced tree has two child nodes. The left node's key value is smaller than the parent node's, and the parent node's key value is smaller than the right node's, and so on. Assuming the number of nodes in the balanced tree is N, the height of the balanced tree is log2(N). When querying data (e.g., player character information) based on a data key value (e.g., player account ID), a hash algorithm is first used to calculate the hash value hash(key). The hash value is then used to locate the head pointer Head(k) of the hash table. The ordered circular linked list located by the head pointer is traversed. The circular linked list is sorted by the data's access popularity, so the node corresponding to the key is first searched in the balanced tree Tree1. The key node is located in the balanced tree, starting from the root node. If the data's key is smaller than the root node's key, the left node is accessed; otherwise, the right node is accessed, and so on. If the key node cannot be found in the current balanced tree, the ordered linked list is returned to the next balanced tree.

[0091] like Figure 6 The following is a diagram of data migration in a specific embodiment. Reading and writing each data key will increase the access frequency of the data. If the access frequency of the data exceeds the counter range of the access frequency of the current tree, the data node needs to be migrated to another balanced tree.

[0092] In order to avoid frequent changes in access frequency, which may cause frequent migration of data nodes and affect performance, data migration detection is only performed when the data migration detection conditions are met; in a specific embodiment, data migration detection is performed after every M data accesses.

[0093] For the adjustment of the M value (the first preset number or the second preset number), an adaptive adjustment scheme is adopted, such as Figure 7 The figure shows a schematic diagram of adaptive adjustment of the M value in a specific embodiment. By counting the time consumed for each data access, the average time consumed for the cumulative access is calculated. If the time consumed exceeds the preset access time threshold of 10ms, it means that the hot data node is not in the hotspot balance tree (such as Figure 6 In this case, the M value needs to be reduced by dividing it by 2 each time to migrate the hotspot data to the hotspot balanced tree as quickly as possible. Once the access time drops below 10ms, the modification of the M value is stopped. It is understandable that in other embodiments, the preset access time threshold can also be set to other times.

[0094] If the M value is too low, frequent data migration will occur. In the above embodiment, the number of data migrations is monitored. If the number exceeds a threshold X, the M value is increased by 1 each time. By continuous adjustment, an appropriate M value is found that satisfies both low data query time and an appropriate number of data migrations.

[0095] The data processing method in the above embodiment effectively improves data search efficiency in games, dynamically detects data read and write popularity, adjusts storage structures, and improves data read and write throughput. By migrating data nodes, high read and write performance of hotspot data is ensured. To prevent frequent hotspot migrations from consuming excessive CPU performance, an adaptive hotspot migration interval adjustment scheme is proposed, ensuring that data migration performance overhead is reduced while minimizing data access time.

[0096] It should be understood that although Figure 2-4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2-4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0097] In one embodiment, Figure 8 As shown, a data processing device is provided. The device can be a software module or a hardware module, or a combination of the two to form a part of a computer device. The device specifically includes: an access count reading module 810, an access count detection module 820 and a data migration module 830, wherein:

[0098] The access count reading module 810 is used to read the access count of each data stored in the key-value storage system. In each hash table of the key-value storage system, data is stored in an ordered linked list, and the access popularity of the data in each linked list node of the ordered linked list decreases in sequence. Each linked list node of the ordered linked list stores a balanced tree. Each balanced tree stores data based on the size of the key value of the data.

[0099] The access count detection module 820 is used to find the target access heat interval corresponding to the access count of the target data when the access count of the target data does not belong to the access heat interval corresponding to the corresponding balanced tree;

[0100] The data migration module 830 is used to migrate the target data to the balanced tree of the linked list node corresponding to the target access popularity interval.

[0101] In the above-mentioned data processing device, in each hash table of the key-value storage system, an ordered linked list is used to store data based on the data access heat, and the data access heat of each linked list node in the ordered linked list decreases in sequence; a balanced tree is stored in each linked list node of the ordered linked list; for each data stored in the key-value storage system, data migration is judged based on the number of accesses, if the number of accesses of the target data does not belong to the access heat interval corresponding to the balanced tree, the target access heat interval corresponding to the number of accesses of the target data is searched, and the target data is migrated to the balanced tree of the linked list node corresponding to the target access heat interval. In the above method, in the key-value storage system, an ordered linked list is used to store data based on the data access popularity, and at the same time, multiple data are stored in each linked list node of the ordered linked list in the form of a balanced tree, and the data is stored based on the key value size in the balanced tree. Therefore, when accessing data, the key value corresponding to the data access request can be found in each balanced tree of the ordered linked list based on the comparison between the key value and the key value size of each node. There is no need to traverse all the data in the hash table, the time required is short, and the reading and writing performance of the data can be improved; and the higher the access popularity of the data, the less search time is required when accessing it.

[0102] In one embodiment, the apparatus further includes: a condition detection module configured to jump to the access count reading module 810 to execute the step of reading the access count of each data stored in the key-value storage system when the data migration detection condition is met.

[0103] Furthermore, in one embodiment, the condition detection module of the above-mentioned device is specifically used to determine that the data migration detection condition is met when the number of accesses to all data in the key-value storage system reaches a first preset number; or, the condition detection module of the above-mentioned device is specifically used to determine that the data migration detection condition is met when the number of accesses to the same data in the key-value storage system reaches a second preset number within a preset time period.

[0104] In one embodiment, the above-mentioned device further includes: an access time acquisition module for acquiring the access time of each data in the key-value storage system; and a preset number adjustment module for adjusting the first preset number or the second preset number based on the access time.

[0105] In another embodiment, the above-mentioned device also includes: a migration number statistics module, which is used to count the number of migrations of each data in the key-value storage system; and a preset number adjustment module, which is used to adjust the first preset number or the second preset number based on the migration number of each data.

[0106] In one embodiment, the above-mentioned device also includes: a data acquisition module, which is used to obtain the data to be stored and perform a hash operation according to the key value of the data to be stored to obtain a corresponding hash value; a positioning module, which is used to locate the hash table header pointer in the key-value storage system according to the hash value; and a data storage module, which is used to store the data to be stored in a target balanced tree in the hash table corresponding to the hash table header pointer based on the key value of the data to be stored; wherein, the data corresponding to the target balanced tree has the lowest access heat.

[0107] In one embodiment, the above-mentioned device also includes a data access request receiving module, which is used to receive the data access request and determine the corresponding hash value according to the key value of the data access request; a hash table search module, which is used to search the corresponding target hash table in the key value storage system according to the hash value; and a search module, which is used to traverse the balanced tree in each linked list node of the target hash table in turn to search for the key value corresponding to the data access request, and when the key value corresponding to the data access request is found in the balanced tree, read the data stored in the node.

[0108] In one embodiment, the key values ​​corresponding to the left node, parent node and right node of the balanced tree are arranged in order of size; in this embodiment, the above-mentioned search module is specifically used to select the current linked list node in the ordered linked list of the target hash table; traverse the balanced tree of the current linked list node starting from the root node; in the balanced tree: if the key value of the access request is less than the key value of the parent node, access the subtree with a key value less than the key value of the parent node; if the key value of the access request is greater than the key value of the parent node, access the subtree with a key value greater than the key value of the parent node; until a node with the same key value as the data access request is found; if the key value of the data access request is not found in the balanced tree of the current linked list node, jump to the balanced tree of the next linked list node for search.

[0109] In one embodiment, the ordered linked list includes an ordered circular linked list.

[0110] In one embodiment, Figure 9 As shown, the above-mentioned device also includes: a request receiving module 910, which is used for receiving a data access request and determining a corresponding hash value according to the key value of the data access request; a hash table positioning module 920, which is used for searching a corresponding target hash table in a key-value storage system according to the hash value; a data search module 930, which is used for sequentially traversing the balanced tree in each linked list node of the target hash table to search for the key value corresponding to the data access request, and when the key value corresponding to the data access request is found in the balanced tree, reading the data stored in the node.

[0111] In one embodiment, the key value corresponding to each child node of the balanced tree is smaller than the key value corresponding to the parent node, and the key value corresponding to the left node of the balanced tree is smaller than the key value corresponding to the right node; in this embodiment, the data search module 930 is specifically used to: in the balanced tree of the current linked list node of the ordered linked list corresponding to the target hash table, compare the key value of the data access request with the size of the key value of each parent node, left node and right node in turn until a node with the same key value as the data access request is found; if the key value of the data access request is not found in the balanced tree of the current linked list node, jump to the balanced tree of the next linked list node in turn for search.

[0112] For the specific definition of the data processing device, please refer to the definition of the data processing method above and will not be repeated here. Each module in the above-mentioned data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0113] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store key-value pair data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0114] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0115] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0116] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0117] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0118] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0119] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Read the number of accesses to each data stored in the key-value storage system; wherein, in each hash table of the key-value storage system, an ordered linked list is used to store data, and the access popularity of data in each linked list node of the ordered linked list decreases in sequence. The hash table is a data structure that is directly accessed based on the key value, and the head pointer of the hash table points to the linked list node corresponding to the data with the highest access popularity in the hash table; a balanced tree is respectively stored in each linked list node of the ordered linked list; each balanced tree stores data based on the size of the key value of the data, and the head pointer of the hash table where the linked list node is located points to the balanced tree corresponding to the data with the highest access popularity; When the number of accesses to the target data does not belong to the access heat interval corresponding to the balanced tree, searching for the target access heat interval corresponding to the number of accesses to the target data; In the balanced tree of the linked list nodes corresponding to the target access heat interval, based on the comparison of the key value of the target data with the key value of the data stored in each linked list node in the balanced tree, the location where the target data needs to be stored in the balanced tree is determined, and the target data is migrated to the location where it needs to be stored in the balanced tree.

2. The data processing method according to claim 1, wherein: When the data migration detection condition is met, the step of reading the access count of each data stored in the key-value storage system is entered.

3. The data processing method according to claim 2, characterized in that: When the number of accesses to all data in the key-value storage system reaches a first preset number, or when the number of accesses to the same data in the key-value storage system within a preset time period reaches a second preset number, it is determined that the data migration detection condition is met.

4. The data processing method according to claim 3, wherein: Also include at least one of the following: The first item, Obtaining access time of each data in the key-value storage system; Adjusting the first preset number or the second preset number based on the access time; The second item, Counting the number of migrations of each data in the key-value storage system; The first preset number of times or the second preset number of times is adjusted based on the migration number of each data.

5. The data processing method according to claim 1, wherein: Also includes: Obtaining data to be stored, and performing a hash operation on the key value of the data to be stored to obtain a corresponding hash value; Locate a hash table header pointer in a key-value storage system according to the hash value; Based on the key value of the data to be stored, storing the data to be stored in a target balanced tree in the hash table corresponding to the hash table header pointer; The data access heat corresponding to the target balanced tree is the lowest.

6. The data processing method according to claim 1, wherein: Also includes: Receive a data access request, and determine a corresponding hash value according to a key value of the data access request; Searching a corresponding target hash table in the key-value storage system according to the hash value; The balanced tree in each linked list node of the target hash table is sequentially traversed to search for the key value corresponding to the data access request. When the key value corresponding to the data access request is found in the balanced tree, the data stored in the node is read.

7. The data processing method according to claim 6, characterized in that: The key values ​​corresponding to the left node, parent node and right node of the balanced tree are arranged in order of size; The step of sequentially traversing the balanced tree in each linked list node of the target hash table to search for the key value corresponding to the data access request includes: Selecting a current linked list node in the ordered linked list of the target hash table; Traverse the balanced tree of the current linked list node starting from the root node; in the balanced tree: if the key value of the access request is less than the key value of the parent node, access the subtree with a key value less than the key value of the parent node; if the key value of the access request is greater than the key value of the parent node, access the subtree with a key value greater than the key value of the parent node; until a node with the same key value as the data access request is found; If the key value of the data access request is not found in the balanced tree of the current linked list node, jump to the balanced tree of the next linked list node to search.

8. The data processing method according to any one of claims 1 to 7, characterized in that: The ordered linked list includes an ordered circular linked list.

9. A data access method, characterized in that: The method comprises: Sending a data access request to a server, the data access request being used to enable the server to determine a corresponding hash value based on a key value of the data access request; searching a corresponding target hash table in a key-value storage system based on the hash value, the target hash table being a data structure that is directly accessed based on the key value; sequentially traversing a balanced tree in each linked list node of the target hash table to search for the key value corresponding to the data access request, and when the key value corresponding to the data access request is found in the balanced tree, reading data to be accessed stored in the linked list node; the data to be accessed belongs to an access popularity interval corresponding to the balanced tree of the linked list node, and a position of the data to be accessed in the balanced tree is determined according to the data processing method according to any one of claims 1 to 8; Receive the data to be accessed fed back by the server.

10. A data processing device, characterized in that: The device comprises: An access count reading module is configured to read the access count of each data item stored in the key-value storage system; wherein, in each hash table of the key-value storage system, an ordered linked list is used to store data, and the access popularity of the data in each linked list node of the ordered linked list decreases in sequence; the hash table is a data structure that is directly accessed based on the key value, and the head pointer of the hash table points to the linked list node corresponding to the data with the highest access popularity in the hash table; a balanced tree is respectively stored in each linked list node of the ordered linked list; each balanced tree stores data based on the size of the key value of the data, and the head pointer of the hash table where the linked list node is located points to the balanced tree corresponding to the data with the highest access popularity; The access count detection module is used to find the target access heat interval corresponding to the access count of the target data when the access count of the target data does not belong to the access heat interval corresponding to the balanced tree; A data migration module is used to determine the required storage location of the target data in the balanced tree of the linked list node corresponding to the target access heat interval based on the comparison of the key value of the target data with the key value of the data stored in each linked list node in the balanced tree, and migrate the target data to the required storage location in the balanced tree.

11. The data processing device according to claim 10, characterized in that The device further comprises: The condition detection module is used to jump to the access count reading module to execute the step of reading the access count of each data stored in the key-value storage system when the data migration detection condition is met.

12. The data processing device according to claim 11, characterized in that The condition detection module is specifically configured to determine that a data migration detection condition is satisfied when the number of accesses to all data in the key-value storage system reaches a first preset number, or when the number of accesses to the same data in the key-value storage system within a preset time period reaches a second preset number.

13. The data processing device according to claim 12, characterized in that The device further comprises: An access time acquisition module, used to acquire the access time of each data in the key-value storage system; The preset number adjustment module is used to adjust the first preset number or the second preset number based on the access time.

14. The data processing device according to claim 12, characterized in that The device further comprises: A migration count statistics module, used to count the number of migrations of each data in the key-value storage system; The preset number adjustment module is configured to adjust the first preset number or the second preset number based on the migration number of each data.

15. The data processing device according to claim 10, characterized in that The device further comprises: A data acquisition module is used to acquire data to be stored and perform a hash operation on the key value of the data to be stored to obtain a corresponding hash value; A positioning module, configured to locate a hash table header pointer in a key-value storage system according to the hash value; The data storage module is used to store the data to be stored in a target balanced tree in the hash table corresponding to the hash table header pointer based on the key value of the data to be stored; the data access heat corresponding to the target balanced tree is the lowest.

16. The data processing device according to claim 10, characterized in that The device further comprises: A data access request receiving module, configured to receive a data access request and determine a corresponding hash value according to a key value of the data access request; A hash table search module, configured to search a corresponding target hash table in the key-value storage system according to the hash value; A search module is used to sequentially traverse the balanced tree in each linked list node of the target hash table to search for the key value corresponding to the data access request, and when the key value corresponding to the data access request is found in the balanced tree, read the data stored in the node.

17. The data processing device according to claim 16, characterized in that The key values ​​corresponding to the left node, parent node and right node of the balanced tree are arranged in order of size; The search module is specifically configured to select a current linked list node in the ordered linked list of the target hash table; and traverse the balanced tree of the current linked list node starting from the root node; In the balanced tree: if the key value of the access request is smaller than the key value of the parent node, access the subtree whose key value is smaller than the key value of the parent node; If the key value of the access request is greater than the key value of the parent node, access the subtree whose key value is greater than the key value of the parent node; Until a node with the same key value as the data access request is found; If the key value of the data access request is not found in the balanced tree of the current linked list node, jump to the balanced tree of the next linked list node to search.

18. The data processing device according to any one of claims 10 to 17, characterized in that: The ordered linked list includes an ordered circular linked list.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images