Calculation management method and device for managing database, storage medium and system
By offloading memory-intensive computations to near-memory devices, the method addresses inefficiencies in Redis, improving performance by reducing data movement and CPU load.
Patent Information
- Application Number
- CN202411287884.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-03
AI Technical Summary
Existing database management systems like Redis face inefficiencies in memory-intensive calculations due to time-consuming operations such as rehashing, traversal of compact lists, and compression/decompression of quick lists, which are addressed through avoidance solutions that impose additional restrictions or functional losses.
The method involves transferring memory-intensive computations to near-memory computing devices, reducing data movement between CPU and memory, and utilizing computing agents and cores to perform these operations efficiently.
This approach significantly improves database efficiency by releasing CPU resources and reducing data movement, thereby enhancing overall performance.
Smart Images

Figure CN119065853A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of database and near-storage computing, and in particular to a computing management method, device, storage medium and system for managing a database. Background Art
[0002] As a cache database for disk databases, the in-memory database Redis is known for its high efficiency. However, there are still several time-consuming calculation scenarios in the processing of data sets of various object types in Redis. In order to ensure performance, Redis has restricted or circumvented it from the software design to avoid excessively long and time-consuming calculations. For example, because rehash is time-consuming, Redis can use progressive rehash, that is, time-sharing to complete it, so that the CPU (Central Processing Unit) will not be blocked for too long. For another example, in the traversal of listpack, Redis can limit the length of listpack from the software. When the length exceeds the threshold, it will be converted to other non-compact data structures for storage. For another example, because the compression and decompression process of quicklist may be time-consuming, Redis supports disabling this function through configuration.
[0003] The idea of these methods is to use avoidance solutions, which do not fundamentally improve the efficiency of these intensive memory access calculations, and these solutions come at the cost of additional restrictions or functional losses. Summary of the invention
[0004] The database computing management method, device, storage medium and system provided by the present disclosure at least solve the above-mentioned technical problems and other technical problems not mentioned above.
[0005] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.
[0006] According to one aspect of the present disclosure, a computing management method for managing a database is provided. The computing management method comprises: based on determining that the computation to be performed includes a preset memory-intensive computation, sending a computing request to a near-memory computing device through a computing agent, wherein the frequency of data transmission between a processor and a memory in the memory-intensive computation is greater than a first preset threshold, and the computing agent is selected to perform the memory-intensive computation; and receiving a notification about a computation result generated by the near-memory computing device based on the computation request through the computing agent.
[0007] Optionally, the memory-intensive computing includes at least one of the following items:
[0008] The memory access frequency of the memory-intensive computing is greater than a second preset threshold;
[0009] The computational simplicity of the memory-intensive computation is less than a third preset threshold;
[0010] The logic control simplicity of the memory-intensive calculation is less than a fourth preset threshold.
[0011] Optionally, the memory-intensive computing includes at least one of the following items:
[0012] Rehashing operations;
[0013] Traversal operations corresponding to compact lists;
[0014] Compress or decompress operations corresponding to quick lists.
[0015] Optionally, the computing agent includes at least one of the following items:
[0016] a first computing agent configured to perform the rehashing operation;
[0017] a second computing agent configured to perform the traversal operation;
[0018] A third computing agent is configured to perform the compression operation or the decompression operation.
[0019] Optionally, the computing management method further includes: executing a computing process based on the computing request by a computing core included in the near memory computing device; wherein the computing core is configured to perform the memory access intensive computing, and the computing core includes at least one of the following items:
[0020] a first computing core configured to perform the rehashing operation;
[0021] a second computing core configured to perform the traversal operation;
[0022] A third computing core is configured to perform the compression operation or the decompression operation.
[0023] Optionally, the performing of the computing process based on the computing request by the computing core comprises: performing the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core; wherein at least one of the process control unit and at least one computing unit in the computing core is configured to perform the memory-intensive computing, and at least one of the process control unit and at least one computing unit in the computing core comprises at least one of the following items:
[0024] a re-hashing control unit and a hashing unit in the first computing core configured to perform the re-hashing operation;
[0025] a traversal control unit, a first decoding unit, and a comparison unit configured to perform the traversal operation in the second computing core;
[0026] A compression control unit, a decompression control unit, a second decoding unit, an encoding unit and a dictionary management unit are configured in the third computing core to perform the compression operation or the decompression operation.
[0027] Optionally, the calculation process is executed based on the calculation request by a process control unit and at least one calculation unit included in the calculation core, including: the re-hash control unit receives a first calculation request corresponding to the re-hash operation, wherein the first calculation request includes hash table information and new bucket size information; the re-hash control unit starts the re-hash operation based on the first calculation request; the re-hash control unit reads the entry addresses of the buckets from the memory in sequence; based on the entry address of the current bucket being empty, the re-hash is terminated; based on the entry address of the current bucket being not empty, the re-hash control unit reads the first entry in the current bucket, and based on the current bucket being empty, switches to read the entry address of the next bucket from the memory; the hash unit recalculates the hash key value according to the new bucket size information; the re-hash control unit reads the entry address of the new bucket; the re-hash control unit re-attaches the entry to the new bucket, uses the next bucket, and re-performs the re-hash operation based on the next bucket; based on the end of the re-hash, the re-hash control unit sends a notification about the calculation result to the calculation agent.
[0028] Optionally, the computing process is executed based on the computing request by a process control unit and at least one computing unit included in the computing core, including: the traversal control unit receives a second computing request corresponding to the traversal operation; the traversal control unit starts the traversal operation based on the second computing request; the traversal control unit reads the current element from the memory in sequence; based on the address of the current element being empty, the traversal operation is ended; based on the address of the current element being not empty, the first decoding unit performs decoding calculation on the current element; the comparison unit compares the current element with the target element; based on determining that the current element is the target element, the traversal operation is ended, and based on determining that the current element is not the target element, the traversal operation is restarted; based on the end of the traversal operation, the traversal control unit sends a notification about the calculation result to the computing agent.
[0029] Optionally, the computing process is executed based on the computing request by a process control unit and at least one computing unit included in the computing core, including: the compression control unit in the near-memory computing device receives a third computing request corresponding to the compression operation, wherein the third computing request includes the address of the original data, the length of the original data, and the address of the compressed data; the compression control unit starts the compression operation based on the third computing request; the compression control unit reads the original data from the memory based on the address of the original data; based on completing the compression target of the original data, the compression operation is ended; the compression control unit controls the encoding unit to generate at least one of the compressed data and the new dictionary code based on the dictionary management unit using the length of the original data as the compression target; the compression control unit writes at least one of the generated compressed data and the new dictionary code to the address of the compressed data, and restarts the compression operation based on the address of the original data; based on ending the compression operation, the compression control unit sends a notification about the calculation result to the computing agent.
[0030] Optionally, the computing process is executed based on the computing request by a process control unit and at least one computing unit included in the computing core, including: the decompression control unit receives a fourth computing request corresponding to the decompression operation, wherein the fourth computing request includes the address of compressed data, the length of compressed data, and the address of decompressed data; the decompression control unit starts the decompression operation based on the fourth computing request; the decompression control unit reads compressed data from the memory based on the address of the compressed data; based on completing the decompression target of the compressed data, the decompression operation is ended; the decompression control unit controls the second decoding unit to generate decompressed data based on the dictionary management unit using the length of the compressed data as the decompression target; the decompression control unit writes the decompressed data to the address of the decompressed data, and restarts the decompression operation based on the address of the compressed data; based on ending the decompression operation, the decompression control unit sends a notification about the calculation result to the computing agent.
[0031] According to another aspect of the present disclosure, a computing management device for managing a database is provided. The computing management device includes: a request sending unit, configured to: send a computing request to a near-memory computing device through a computing agent based on determining that the computing to be performed includes a preset memory-intensive computing, wherein the frequency of data transmission between the processor and the memory in the memory-intensive computing is greater than a first preset threshold, and the computing agent is selected to perform the memory-intensive computing; a notification receiving unit, configured to: receive a notification about a computing result generated by the near-memory computing device based on the computing request through the computing agent.
[0032] Optionally, the memory-intensive computing includes at least one of the following items:
[0033] The memory access frequency of the memory-intensive computing is greater than a second preset threshold;
[0034] The computational simplicity of the memory-intensive computation is less than a third preset threshold;
[0035] The logic control simplicity of the memory-intensive calculation is less than a fourth preset threshold.
[0036] Optionally, the memory-intensive computing includes at least one of the following items:
[0037] Rehashing operations;
[0038] Traversal operations corresponding to compact lists;
[0039] Compress or decompress operations corresponding to quick lists.
[0040] Optionally, the computing agent includes at least one of the following items:
[0041] a first computing agent configured to perform the rehashing operation;
[0042] a second computing agent configured to perform the traversal operation;
[0043] A third computing agent is configured to perform the compression operation or the decompression operation.
[0044] Optionally, the near memory computing device is configured to: perform a computing process based on the computing request by a computing core included in the near memory computing device; wherein the computing core is configured to perform the memory access intensive computing, and the computing core includes at least one of the following items:
[0045] a first computing core configured to perform the rehashing operation;
[0046] a second computing core configured to perform the traversal operation;
[0047] A third computing core is configured to perform the compression operation or the decompression operation.
[0048] Optionally, the near-memory computing device is configured to: execute the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core; wherein at least one of the process control unit and at least one computing unit in the computing core is configured to perform the memory-access intensive computing, and at least one of the process control unit and at least one computing unit in the computing core includes at least one of the following items:
[0049] a re-hashing control unit and a hashing unit in the first computing core configured to perform the re-hashing operation;
[0050] a traversal control unit, a first decoding unit, and a comparison unit configured to perform the traversal operation in the second computing core;
[0051] A compression control unit, a decompression control unit, a second decoding unit, an encoding unit and a dictionary management unit are configured in the third computing core to perform the compression operation or the decompression operation.
[0052] Optionally, the near memory computing device is configured to: receive a first computing request corresponding to the re-hash operation through the re-hash control unit, wherein the first computing request includes hash table information and new bucket size information; the re-hash control unit starts the re-hash operation based on the first computing request; the re-hash control unit reads the entry addresses of the buckets from the memory in sequence; based on the entry address of the current bucket being empty, ends the re-hash operation; based on the entry address of the current bucket being not empty, the re-hash control unit reads the first entry in the current bucket, and based on the current bucket being empty, switches to read the entry address of the next bucket from the memory; the hash unit recalculates the hash key value according to the new bucket size information; the re-hash control unit reads the entry address of the new bucket; the re-hash control unit re-attaches the entry to the new bucket, uses the next bucket, and re-performs the re-hash operation based on the next bucket; based on the end of the re-hash operation, the re-hash control unit sends a notification about the calculation result to the computing agent.
[0053] Optionally, the near memory computing device is configured to: receive a second computing request corresponding to the traversal operation through the traversal control unit; the traversal control unit starts the traversal operation based on the second computing request; the traversal control unit reads the current element from the memory in sequence; based on the address of the current element being empty, ends the traversal operation; based on the address of the current element being not empty, the first decoding unit performs decoding calculation on the current element; the comparison unit compares the current element with the target element; based on determining that the current element is the target element, ends the traversal operation, and based on determining that the current element is not the target element, restarts the traversal operation; based on ending the traversal operation, the traversal control unit sends a notification about the calculation result to the computing agent.
[0054] Optionally, the near-memory computing device is configured to: receive a third computing request corresponding to the compression operation through the compression control unit, wherein the third computing request includes the address of the original data, the length of the original data, and the address of the compressed data; the compression control unit starts the compression operation based on the third computing request; the compression control unit reads the original data from the memory based on the address of the original data; based on completing the compression target of the original data, ends the compression operation; the compression control unit controls the encoding unit to generate at least one of the compressed data and the new dictionary code based on the dictionary management unit using the length of the original data as the compression target; the compression control unit writes at least one of the generated compressed data and the new dictionary code into the address of the compressed data, and restarts the compression operation based on the address of the original data; based on ending the compression operation, the compression control unit sends a notification about the calculation result to the computing agent.
[0055] Optionally, the near memory computing device is configured to: receive a fourth computing request corresponding to the decompression operation through the decompression control unit, wherein the fourth computing request includes the address of the compressed data, the length of the compressed data, and the address of the decompressed data; the decompression control unit starts the decompression operation based on the fourth computing request; the decompression control unit reads the compressed data from the memory based on the address of the compressed data; based on completing the decompression target of the compressed data, ends the decompression operation; the decompression control unit controls the second decoding unit to generate decompressed data based on the dictionary management unit using the length of the compressed data as the decompression target; the decompression control unit writes the decompressed data to the address of the decompressed data, and restarts the decompression operation based on the address of the compressed data; based on ending the decompression operation, the decompression control unit sends a notification about the calculation result to the computing agent.
[0056] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the computing management method for managing a database as described above.
[0057] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the computing management method for managing a database as described above.
[0058] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0059] According to the computing management method, device, storage medium and system of the database disclosed in the present invention, memory-intensive computing can be transferred from the database to computing memory such as near memory computing (Processing Near Memory, PNM) devices, which can significantly reduce the data movement between the CPU and memory in the database, release the CPU computing resources of the database, and thus fundamentally improve the efficiency of the database. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0061] Figure 1 A schematic diagram showing the operation process of a progressive rehashing operation according to an exemplary embodiment of the present disclosure;
[0062] Figure 2 A schematic diagram showing data structure conversion of a compact list according to an exemplary embodiment of the present disclosure;
[0063] Figure 3 A schematic diagram showing enabling / disabling compression of a quick list configuration according to an exemplary embodiment of the present disclosure;
[0064] Figure 4 A schematic diagram showing data transmission of a rehashing process according to an exemplary embodiment of the present disclosure;
[0065] Figure 5 A schematic diagram showing data transmission during a process of traversing a compact list according to an exemplary embodiment of the present disclosure;
[0066] Figure 6 A data transmission schematic diagram showing a compression / decompression quick list process according to an exemplary embodiment of the present disclosure;
[0067] Figure 7 A flowchart showing a database computing management method according to an exemplary embodiment of the present disclosure;
[0068] Figure 8 A data interaction diagram showing a database computing management method according to an exemplary embodiment of the present disclosure;
[0069] Fig. 9 An interactive schematic diagram showing a rehashing core of an exemplary embodiment of the present disclosure;
[0070] Fig.10 An interactive schematic diagram showing a traversal core of an exemplary embodiment of the present disclosure;
[0071] Fig.11 A schematic diagram showing the interaction of the compression / decompression core of an exemplary embodiment of the present disclosure;
[0072] Fig.12 A timing diagram showing a re-hash calculation process of an exemplary embodiment of the present disclosure;
[0073] Fig.13 A timing diagram showing a process of traversing a compact list calculation according to an exemplary embodiment of the present disclosure;
[0074] Fig.14 A timing diagram showing a compression / decompression fast list calculation process of an exemplary embodiment of the present disclosure;
[0075] Fig.15 A schematic diagram showing a comparison between a related technical solution and an exemplary embodiment of the present disclosure;
[0076] Fig.16 A block diagram showing a database computing management device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0077] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0079] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0080] A database (such as Redis) can be a cache database as a disk database. From the perspective of physical resources, Redis can use memory-saving data structures to store data to save memory; from the perspective of efficiency, Redis's single-threaded feature enables Redis to run as efficiently as possible and avoid blocking. In the process of processing data sets of various object types in Redis, there are several time-consuming calculation scenarios. In order to ensure efficient or consistent performance, Redis has restricted or circumvented them in software design to avoid excessively long and time-consuming calculations.
[0081] For example, a dictionary (which can be called DICT) can be an example of a widely used data structure in Redis. Hash, set, ordered set in Redis objects, and database dictionaries maintained by Redis itself all use DICT data structures for storage.
[0082] Rehash operation refers to an operation used to dynamically expand and shrink the dictionary. As the number of elements changes, Redis can maintain the load factor within a reasonable range through rehash operations, dynamically adjust the bucket size, recalculate the key value (such as hash key value) of each entry according to the new bucket size, and complete the mounting to achieve a balance between memory and efficiency.
[0083] Because Rehash is time-consuming, Redis can use progressive Rehash. For example, Rehash can be completed in time-sharing to avoid CPU blocking for too long. When Rehash is needed, the server does not perform Rehash on all key-value pairs in the hash table (hashtable, ht) at once, such as rehashing from ht[0] to ht[1]. Instead, it performs Rehash in multiple times and progressively: each time an add, delete, search, or update operation is performed on the dictionary, all entries in a bucket are rehashed.
[0084] Figure 1A schematic diagram showing the operation process of a progressive rehashing operation according to an exemplary embodiment of the present disclosure is shown.
[0085] Reference Figure 1 , when the goal is to increase the bucket size from 4 to 8, bucket[0] and bucket[2] can be rehashed; the example operation of the rehashing operation is described as follows:
[0086] Step 1: When a user's "get" operation occurs, rehash all entries in bucket[0];
[0087] Step 2: When a user's "set" operation occurs, rehash all entries in bucket[1];
[0088] Step 3: When a user's "update" operation occurs, rehash all entries in bucket[3];
[0089] Step 4: The rehash process ends.
[0090] In the above process, the Rehash cycle will be extended, and user access will also require additional resources for calculation.
[0091] For another example, listpack (also called compact list) is a data structure that saves memory by arranging data compactly in memory. Listpack can be used for storage in the following scenarios: OBJ_HASH, OBJ_ZSET, and OBJ_LIST data storage.
[0092] When performing basic operations such as searching, inserting, and deleting on the data in the listpack, the data area in the entire listpack is traversed and related calculation operations are performed. Since the listpack can be compactly stored in the memory, frequent traversal of the listpack will result in large memory consumption. In redis, in order to reduce the computational consumption when traversing the listpack, the software solution may be to limit the length of the listpack. For example, when the length of the listpack exceeds a threshold, the listpack will be converted to other non-compact data structures for storage. An example of conversion rules between data structures according to the embodiment is shown in Table 1 below:
[0093] Table 1
[0094]
[0095] Figure 2 A schematic diagram showing data structure conversion of a compact list according to an exemplary embodiment of the present disclosure.
[0096] Reference Figure 2 , elm may refer to the entry in listpack; encodeType may refer to the encoding type; backlen may refer to the offset of the backtracking start address length of the current entry, that is, the entry length during reverse traversal; EOF may refer to the end mark of listpack.
[0097] To insert a new entry into a listpack, when the length of the entry in the listpack is greater than a threshold, additional calculations may be used to convert the listpack into a non-compactly arranged data structure.
[0098] For example, in the storage scenario where the encoding type is OBJ_ZSET, listpack is converted to skiplist, and Node is the basic data unit of skiplist; in the storage scenario where the encoding type is OBJ_HASH, listpack is converted to DICT, and dictEntry is the basic data unit of DICT; in the storage scenario where the encoding type is OBJ_LIST, listpack is converted to quicklist, and qlNode is the basic data unit of quicklist, head refers to the header of quicklist, and tail refers to the tail of quicklist.
[0099] For another example, quicklist can be an underlying data structure of type OBJ_LIST. In order to save space, Redis can support LZF compression (LZF, a compression algorithm) for quicklist. For example, all intermediate nodes except the two nodes of the head and tail nodes are stored with LZF compression; when accessing the intermediate nodes, decompression can be performed first; when saving the intermediate nodes, compression can be performed first.
[0100] Since the compression and decompression process is time-consuming, Redis supports disabling this function by configuring the compression enable (compressionenable) / compression disable (compressiondisable) switch.
[0101] Figure 3 A schematic diagram showing enabling / disabling compression for a quick list configuration according to an exemplary embodiment of the present disclosure is shown.
[0102] Reference Figure 3When compression is enabled, compressing the nodes in the quicklist can save memory, but it will also cause high CPU (Central Processing Unit) consumption in the process of accessing and saving the quicklist. However, when compression is disabled, the nodes in the quicklist exist in the form of original data, which can reduce CPU calculations, but will occupy more memory and waste storage space.
[0103] Some methods that perform high CPU consumption calculations may use the above workarounds, but they do not fundamentally improve the efficiency of these intensive memory access calculations, and they still come at the cost of additional restrictions or functional losses. The workarounds will cause other side effects and cannot really improve the performance of Redis. For example, the problems with the above workarounds can be shown in the following table:
[0104]
[0105] In order to fundamentally improve the efficiency of Redis, the calculation mode of the above time-consuming calculation can be analyzed as follows:
[0106] Figure 4 A schematic diagram of data transmission between devices during a rehashing process according to an exemplary embodiment of the present disclosure is shown.
[0107] Reference Figure 4 For the rehashing process, in one rehashing, it is necessary to traverse all entries in the hash table. If there are M buckets in the hash table and N entries in each bucket (both M and N are integers greater than zero), then between the host CPU and the memory, the host CPU accesses the memory M×N times to read the entry, and after reading each entry, the host CPU performs hash calculation to obtain a new hash key value, and finally writes the obtained new hash key value to the hash table in the memory through memory access again. The overall pattern of this rehash process can be summarized as the RCW (Read-Calculate-Write) pattern.
[0108] Figure 5 A schematic diagram of data transmission between devices showing a process of traversing a compact list according to an exemplary embodiment of the present disclosure.
[0109] Reference Figure 5For the traversal process of listpack, the host CPU can read each element in the listpack from the memory in turn. If there are N elements in the listpack, the host CPU can access the memory N times and read N times. The overall mode of the traversal process of this listpack can be summarized as the RC (Read-Calculate) mode.
[0110] Figure 6 A data transmission diagram showing a compression / decompression quick list process according to an exemplary embodiment of the present disclosure.
[0111] Reference Figure 6 For the compression and / or decompression process of the quick list, the host CPU can access the memory in sequence to retrieve the data from the memory, and after performing the compression / decompression calculation process in the host CPU, access the memory to write the data back to the memory. If N quick lists are processed, the above process can be repeated N times. The overall mode of the compression / decompression process of this quick list can be summarized as the RCW (Read-Calculate-Write) mode.
[0112] The above analysis shows that the above computing scenarios are all memory-intensive computing, and because the computing process requires multiple accesses to the memory, it will also consume a lot of time. These computing scenarios have the following common characteristics:
[0113] For example, in terms of memory access mode, frequent data transfer can be performed in these computing scenarios, and basically repeated in two modes: RC mode and RCW mode;
[0114] For another example, in terms of computing characteristics, in these computing modes the computing logic may be relatively simple, and the same computing process can be repeatedly executed without the need for complex logic control.
[0115] In order to solve the above problems, the present disclosure provides a database computing management method, device, storage medium and system, which can transfer time-consuming memory-intensive calculations from the database to a place close to the storage, such as to a computational memory such as a near-storage computing device, which can significantly reduce the data movement between the CPU and memory in the database, release the CPU computing resources of the database, and thus fundamentally improve the efficiency of the database.
[0116] Next, we will refer to Figures 7 to 16 The computing management method, device, storage medium and system of the database disclosed in the present invention are specifically described.
[0117] Reference Figure 7In step 701, based on determining that the computation to be performed is a preset memory-intensive computation, a corresponding computation request may be sent to a near-memory computing device through a computation agent. In an embodiment, the memory-intensive computation may be a computation in which the frequency of data transmission between the processor and the memory is greater than a first preset threshold, and a computation agent may be selected to perform the memory-intensive computation.
[0118] According to an exemplary embodiment of the present disclosure, a user can perform data processing operations such as search, insert, update, etc. on a database such as Redis. In the process of executing the current user operation, the database can judge whether the calculation to be performed in the current step belongs to a preset memory-intensive calculation. Memory-intensive calculations can be calculations in which the frequency of data transmission between the processor (e.g., CPU) and the memory is greater than a first preset threshold. For example, this can indicate that the calculation process of this calculation type belongs to a calculation type with a high frequency of data transmission between the processor and the memory, such as a calculation type that completes data transmission by frequently repeating the "RC" and "RCW" modes; based on determining that the calculation to be performed is a preset memory-intensive calculation, a corresponding calculation request can be sent to a near memory computing device (Processing Near Memory device, PNM device) through a computing agent configured to perform memory-intensive calculations to avoid directly performing preset memory-intensive calculations in the database.
[0119] According to an exemplary embodiment of the present disclosure, for non-memory access intensive computing, such as a computing mode without frequent data transmission between the processor and the memory, specifically, the frequency of data transmission between the processor and the memory is less than or equal to a first preset threshold value to indicate non-memory access intensive computing, and computing offloading may not be performed. For example, the computing process of non-memory access intensive computing may not be transferred from the database to the near-memory computing device, but the corresponding computing process may still be completed through data transmission between the host and the memory of the database.
[0120] According to an exemplary embodiment of the present disclosure, memory-intensive computing may also include at least one of the following items: computing whose frequency of accessing memory (memory access frequency) is greater than a second preset threshold; computing whose computational simplicity is less than a third preset threshold; computing whose logic control simplicity is less than a fourth preset threshold.
[0121] According to an exemplary embodiment of the present disclosure, the type of memory-intensive calculation can also be determined by at least one of the following items: a calculation in which the frequency of accessing memory is greater than a second preset threshold, which can be used to indicate that the current calculation type belongs to a calculation type that requires frequent data access to the memory; a calculation in which the simplicity of calculation is less than a third preset threshold, which can be used to indicate that the current calculation type belongs to a calculation type that can be completed by simply executing the same calculation process multiple times; a calculation in which the simplicity of logical control is less than a fourth preset threshold, which can be used to indicate that the current calculation type belongs to a calculation type that only requires simple logical control and does not require the addition of complex algorithms; in addition, it can also include a calculation type in which the final result of the calculation has a small memory requirement, which can be used to indicate that the calculation result of the current calculation type belongs to a calculation type that does not require a large amount of data storage in the memory.
[0122] According to an exemplary embodiment of the present disclosure, memory-intensive computing may include at least one of the following items: a rehashing operation or process; a traversal operation or process corresponding to a compact list; a compression operation or process corresponding to a fast list or a decompression operation or process corresponding to a fast list.
[0123] According to an exemplary embodiment of the present disclosure, through the above analysis of the characteristics of the computing process, at least one of the following items can be classified as memory-intensive computing: for example, rehashing, traversing a compact list, and compressing / decompressing a quick list.
[0124] According to an exemplary embodiment of the present disclosure, the computing agent selected or determined in the database may include at least one of the following items: a first computing agent configured to perform a rehashing operation; a second computing agent configured to perform a traversal of a compact list; and a third computing agent configured to perform compression / decompression of a quick list.
[0125] According to an exemplary embodiment of the present disclosure, the corresponding computing agent can be configured to perform different memory-intensive calculations in the database. For example, the computing agent can include at least one of the following items: a first computing agent configured to perform a rehashing operation, which can also be represented by a rehashing agent; a second computing agent configured to perform a traversal of a compact list, which can also be represented by a traversal agent; a third computing agent configured to perform compression / decompression of a quick list, which can also be represented by a compression / decompression agent.
[0126] At step 702 , a notification regarding a computation result generated by a near-storage computing device based on a computation request may be received through a computation agent.
[0127] According to an exemplary embodiment of the present disclosure, the above method may also include: executing a corresponding computing process based on a computing request through a computing core in a near-memory computing device; wherein the computing core is configured to perform memory-intensive computing, and the computing core includes at least one of the following items: a first computing core configured to perform a re-hash operation; a second computing core configured to perform a traversal of a compact list; and a third computing core configured to perform compression / decompression of a quick list.
[0128] According to an exemplary embodiment of the present disclosure, a computing core can be selected or determined in a near-memory computing device for memory-intensive computing, and a corresponding computing process can be executed by the computing core based on a computing request sent by a database; wherein, a corresponding computing core can be selected or determined in a near-memory computing device for different memory-intensive computing, for example, the computing core includes at least one of the following items: a first computing core configured to perform a rehashing operation, which can be represented by a rehashing core; a second computing core configured to perform a traversal of a compact list, which can be represented by a traversal core; a third computing core configured to perform a compression / decompression quick list, which can be represented by a compression / decompression core.
[0129] According to an exemplary embodiment of the present disclosure, a corresponding computing process can be executed based on a computing request by a computing core in a near-memory computing device, which can specifically include: executing the corresponding computing process based on a computing request by a process control unit and at least one computing unit in the computing core; wherein, at least one of the process control unit and at least one computing unit in the computing core is selected or determined for memory-intensive computing, and at least one of the process control unit and at least one computing unit in the computing core can include at least one of the following items: a re-hash control unit and a hash unit configured to perform a re-hash operation in a first computing core; a traversal control unit, a first decoding unit, and a comparison unit configured to perform a traversal operation in a second computing core; a compression / decompression control unit, a second decoding unit, an encoding unit, and a dictionary management unit configured to perform a compression operation or a decompression operation in a third computing core. In an embodiment, the compression / decompression control unit may include or be referred to as at least one of a compression control unit and a decompression control unit.
[0130] According to an exemplary embodiment of the present disclosure, a corresponding computing process can be executed based on a computing request by a computing core in a near-memory computing device. For example, a process control unit and at least one computing unit with a built-in corresponding algorithm can be set in the computing core according to the computing process characteristics of different memory-intensive computing to execute the corresponding computing process based on the computing request. For example, a re-hash control unit and a hash unit for re-hash operations can be included in the first computing core; a traversal control unit, a first decoding unit, and a comparison unit for traversing a compact list can be included in the second computing core; and a compression / decompression control unit, a second decoding unit, an encoding unit, and a dictionary management unit for compressing / decompressing a quick list can be included in the third computing core.
[0131] According to an exemplary embodiment of the present disclosure, the computing agent on the database host side may be responsible for communicating with the computing core on the near-memory computing device to initiate memory-intensive computing, generate and transmit the parameters required for the corresponding memory-intensive computing, collect the computing results and subsequent logical processing. For re-hash operations, the re-hash agent may detect the conditions of the re-hash operation, calculate the new bucket size, and initiate a re-hash request to the computing core. For traversing a compact list, the traversal agent may initiate a request to traverse the compact list to the computing core, transmit conditional parameters, perform subsequent processing and exception handling on the returned results, etc. For compression / decompression of quick lists, the compression / decompression agent may initiate a compression / decompression request to the computing core, transmit data addresses, maintain subsequent logic and exception handling, etc.
[0132] According to an exemplary embodiment of the present disclosure, in the computing core in the near-memory computing device, the control unit may be responsible for interacting with the computing agent on the host side, calling each computing unit to execute the computing process, and accessing the memory in the near-memory computing device to read and write data.
[0133] Figure 8 A data interaction diagram showing a database computing management method according to an exemplary embodiment of the present disclosure.
[0134] Reference Figure 8According to an exemplary embodiment of the present disclosure, according to the above-mentioned database computing management method, a computing agent for access-intensive computing can be set in the management system of the Redis database executed by the host CPU, and one or more processing flows of one or more access-intensive computing can be set. For example, corresponding computing agents can be set in the computing agent for different memory-intensive computing. According to an embodiment, a first computing agent can be set for rehashing operations, such as a rehashing agent in the management system of the Redis database; a second computing agent can be set for traversing a compact list, such as a traversal agent in the management system of the Redis database; a third computing agent can be set for compressing / decompressing a quick list, such as a compression / decompression agent in the management system of the Redis database.
[0135] Correspondingly, a computing core for memory-intensive computing can be set in the near-memory computing device to execute the corresponding computing process in response to the computing request sent by the computing agent. According to an embodiment, a first computing core can be set for rehashing operations, such as a rehashing core in a computing core; a second computing core can be set for traversing a compact list, such as a traversal core in a computing core; and a third computing core can be set for compressing / decompressing a fast list, such as a compression / decompression core in a computing core.
[0136] According to the embodiment, a process control unit and at least one computing unit may be set in each computing core according to the computing process characteristics of different memory-intensive computing to execute the corresponding computing process in response to the computing request. For example, a re-hash control unit and a hash unit may be set in a re-hash core for re-hash operations; a traversal control unit, a first decoding unit, and a comparison unit may be set in a traversal core for traversing a compact list; and a compression / decompression control unit, a second decoding unit, an encoding unit, and a dictionary management unit may be set in a compression / decompression core for compressing / decompressing a fast list.
[0137] Fig. 9 A schematic diagram showing the interaction of a rehashing core according to an exemplary embodiment of the present disclosure.
[0138] According to an exemplary embodiment of the present disclosure, in combination with reference Figure 8 and Fig. 9 For the re-hash operation, a re-hash request can be sent to the re-hash control unit through the re-hash proxy, and the re-hash control unit calls the hash unit to perform the re-hash calculation process and access the memory in the near-storage computing device to read and write data, and the re-hash proxy receives the calculation result sent by the re-hash control unit.
[0139] According to an exemplary embodiment of the present disclosure, specifically, the re-hash control unit may be used to execute a re-hash process of the entire hash table, and the hash unit may be a built-in hash algorithm.
[0140] Fig.10 A schematic diagram showing an interaction of a traversal core according to an exemplary embodiment of the present disclosure.
[0141] According to an exemplary embodiment of the present disclosure, in combination with reference Figure 8 and Fig.10 For traversing the compact list, a request for traversing the compact list can be sent to the traversal control unit through the traversal agent, and the traversal control unit calls the first decoding unit and the comparison unit to execute the calculation process of traversing the compact list, and accesses the memory in the near storage computing device to read and write data, and the traversal agent receives the calculation results sent by the traversal control unit.
[0142] According to an exemplary embodiment of the present disclosure, specifically, the traversal control unit can be used to execute the traversal process of the list package under specified conditions, and this specified condition can indicate that when the user operation needs to traverse the compact list; the first decoding unit can be a built-in fast list decoding algorithm; the comparison unit can be a series of built-in data comparison algorithms.
[0143] Fig.11 A schematic diagram showing the interaction of the compression / decompression core of an exemplary embodiment of the present disclosure.
[0144] According to an exemplary embodiment of the present disclosure, in combination with reference Figure 8 and Fig.11 For the compression / decompression quick list, a compression / decompression request can be sent to the compression / decompression control unit through the compression / decompression agent, and the compression / decompression control unit calls the second decoding unit, the encoding unit and the dictionary management unit to perform the calculation process of the compression / decompression quick list, and access the memory in the near storage computing device to read and write data, and the compression / decompression agent receives the calculation result sent by the compression / decompression control unit. Among them, the dictionary management unit can access the memory in the near storage computing device alone to read and write data.
[0145] According to an exemplary embodiment of the present disclosure, specifically, the compression / decompression control unit can be used to perform a compression and / or decompression process on data of a specified length; the second decoding unit can be a built-in decoding algorithm for a file type (e.g., ZLF); the encoding unit can be or can execute a built-in ZLF encoding algorithm; and the dictionary management unit can be used to maintain a ZLF dictionary. It can be understood that the above compression and / or decompression process is not limited to processing ZLF files.
[0146] In addition, return to reference Figure 8, the non-memory-intensive computing in the management system of the Redis database executed by the host CPU can directly interact with the memory in the near-memory computing device to read and write data, without setting up a computing agent in the Redis database host and / or a corresponding computing core in the near-memory computing device. The above non-memory-intensive computing may include but is not limited to list inserts, hash lookups, etc.
[0147] It is understandable that memory-intensive computing and non-memory-intensive computing are not limited to the above-listed types of computing.
[0148] According to an exemplary embodiment of the present disclosure, a corresponding computing process may be executed in response to a computing request by a process control unit and at least one computing unit in a computing core. According to an embodiment, such a calculation process may include: a re-hash control unit in a near-memory computing device receives a first calculation request corresponding to a re-hash operation, wherein the first calculation request includes hash table information and new bucket size information; the re-hash control unit starts to perform a re-hash operation based on the first calculation request; the re-hash control unit reads the entry addresses of the buckets from the memory in sequence; when the entry address of the current bucket is empty, the re-hash operation is terminated; when the entry address of the current bucket is not empty, the re-hash control unit reads the first entry in the current bucket, and when the current bucket is empty, switches to read the entry address of the next bucket from the memory; the hash unit recalculates the hash key value according to the new bucket size information; the re-hash control unit reads the entry address of the new bucket; the re-hash control unit re-attaches the entry to the new bucket, uses the next bucket (for example, uses the next bucket of the current bucket as the new current bucket), and re-performs the re-hash operation based on the next bucket (for example, re-starts from reading the first entry in the current bucket); when the re-hash operation is terminated, the re-hash control unit sends a notification about the calculation result to the calculation agent of the database.
[0149] Fig.12 A timing diagram showing a re-hash operation calculation process according to an exemplary embodiment of the present disclosure.
[0150] Reference Fig.12 According to an exemplary embodiment of the present disclosure, the aforementioned near-memory computing device may be a near-memory computing device based on Compute Express Link (CXL), which is a high-speed serial computer expansion bus standard, so that the near-memory computing device based on CXL can be added / deleted online on the CXL server according to the business volume by using the hot-plug function to perform dynamic capacity adjustment without causing business interruption. It can be represented by CXL-PNM.
[0151] According to an exemplary embodiment of the present disclosure, for the calculation process of rehashing, on the database host side, the CPU can control the rehashing agent to send a first calculation request (rehashing request) corresponding to the rehashing to the rehashing control unit in the near-storage computing device, wherein the first calculation request can carry hash table information, new bucket size information, etc., so that the rehashing control unit can start to perform rehashing through the first calculation request; in the process of the rehashing control unit performing rehashing, the rehashing control unit can read the entry address of the bucket from the memory in sequence according to the hash table information and the new bucket size information; for the current bucket, the first read bucket can be used as the old bucket (represented by bucket_old[i] in FIG12 ), and the later read bucket can be used as the new bucket ( Fig.12 In the example, bucket_new is used to represent the new bucket. After reading the current bucket, the entry in the current bucket can be read (entry of bucket_old[i]). When the entry address of the current bucket is empty (or indicates that all buckets in the hash table have been read, last bucketdone), the rehashing operation ends. When the entry address of the current bucket is not empty (or indicates that all buckets in the hash table have not been read), the rehashing control unit reads the first entry in the current bucket. When the current bucket is empty (or indicates that all entries in the bucket have been read, last bucketdone), the rehashing control unit ends. entrydone), switches to read the entry address of the next bucket from the memory; the re-hash control unit calls the hash unit to recalculate the hash key value according to the new bucket size information and writes the hash key value into the memory; the re-hash control unit reads the entry address of the new bucket; the re-hash control unit re-attaches the entry (entry) to the new bucket (bucket_new), takes the next bucket of the current bucket as the new current bucket, and starts execution again from reading the first entry in the current bucket; when the re-hash is finished, the re-hash control unit can send a calculation result notification to the re-hash agent of the database.
[0152] According to an exemplary embodiment of the present disclosure, a corresponding computing process can be executed in response to a computing request by a process control unit and at least one computing unit in a computing core. This step may include: a traversal control unit in a near-memory computing device receives a second computing request corresponding to a traversal of a compact list; the traversal control unit starts to traverse the compact list based on the second computing request; the traversal control unit reads the current element from the memory in sequence; when the address of the current element is empty, the traversal of the compact list is terminated; when the address of the current element is not empty, the first decoding unit performs a decoding calculation on the current element; the comparison unit compares whether the current element is the target element, and if so, terminates the traversal of the compact list, and if not, re-reads the current element in sequence from the memory to start execution; when the traversal of the compact list is terminated, the traversal control unit sends a calculation result notification to the computing agent of the database.
[0153] Fig.13 A timing diagram showing a calculation process of traversing a compact list according to an exemplary embodiment of the present disclosure.
[0154] Reference Fig.13 According to an exemplary embodiment of the present disclosure, for the calculation process of traversing the compact list, on the database host side, the CPU can control the traversal agent to send a second calculation request (a traversal compact list request) corresponding to the traversal compact list to the traversal control unit in the near-storage computing device, so that the traversal control unit can start to traverse the compact list through the second calculation request; in the process of the traversal control unit traversing the compact list, the traversal control unit can read the current element (element[i]) from the memory in sequence; when the address of the current element is empty, the traversal compact list is terminated; when the address of the current element is not empty, the traversal control unit calls the first decoding unit to perform decoding calculation on the current element; the traversal control unit calls the comparison unit to compare whether the current element is the target element ( Fig.13 If yes, the traversal of the compact list is terminated; if no, the current elements are read from the memory again to start execution; when the traversal of the compact list is terminated, the traversal control unit can send a calculation result notification to the traversal agent of the database.
[0155] According to an exemplary embodiment of the present disclosure, the traversal agent may directly read or write the memory in the near-storage computing device to exchange data.
[0156] According to an exemplary embodiment of the present disclosure, a corresponding computing process can be executed in response to a computing request by a process control unit and at least one computing unit in a computing core, and this step may include: a compression control unit in a near-memory computing device receives a third computing request corresponding to a compression quick list, wherein the third computing request includes an address of original data, a length of original data, and an address of compressed data; the compression control unit starts to execute the compression quick list based on the third computing request; the compression control unit reads the original data from the memory based on the address of the original data; based on completing the compression target of the original data, the execution of the compression quick list is terminated; the compression control unit controls the encoding unit to generate at least one of compressed data and a new dictionary code based on the dictionary management unit with the length of the original data as the compression target; the compression control unit writes at least one of the generated compressed data and the new dictionary code to the address of the compressed data, and restarts the compression operation based on the address of the original data (for example, re-reads the original data from the memory based on the address of the original data to start execution); based on the termination of the execution of the compression quick list, the compression control unit sends a notification about the calculation result to the computing agent of the database.
[0157] According to an exemplary embodiment of the present disclosure, a corresponding computing process can be executed in response to a computing request by a process control unit and at least one computing unit in a computing core, and this step may include: a decompression control unit in a near-memory computing device receives a fourth computing request corresponding to a decompression quick list, wherein the fourth computing request includes an address of compressed data, a length of compressed data, and an address of decompressed data; the decompression control unit starts to execute the decompression quick list based on the fourth computing request; the decompression control unit reads compressed data from a memory based on the address of compressed data; based on completing the decompression target of the compressed data, the execution of the decompression quick list is terminated; the decompression control unit controls the second decoding unit to generate decompressed data based on the dictionary management unit with the length of compressed data as the decompression target; the decompression control unit writes the generated decompressed data to the address of the decompressed data, and restarts the decompression operation based on the address of the compressed data (for example, re-reading the compressed data from the memory based on the address of the compressed data to start execution); based on the termination of the execution of the decompression quick list, the decompression control unit sends a calculation result notification to the computing agent of the database.
[0158] Fig.14 A timing diagram showing a calculation process of a compression operation or a decompression operation according to an exemplary embodiment of the present disclosure.
[0159] Reference Fig.14According to an exemplary embodiment of the present disclosure, for the calculation process of the compressed quick list, on the database host side, the host CPU can control the compression agent to send a third calculation request (compression request) corresponding to the compressed quick list to the compression control unit in the near-storage computing device, so that the compression control unit can start to execute the compressed quick list through the third calculation request, wherein the third calculation request can carry the address of the original data, the length of the original data, and the address of the compressed data; in the process of the compression control unit executing the compressed quick list, the compression control unit can read N bytes of original data from the memory based on the address of the original data, and the byte length can be determined by the information of the length of the original data carried in the third calculation request. determined; when the compression target of the original data has been achieved, the execution of the compression quick list is terminated; when the compression target of the original data has not been achieved, the compression control unit can call the dictionary management unit to perform dictionary comparison on the original data, call the encoding unit to compress and encode the original data with the length of the original data as the compression target to generate compressed data, or generate a new dictionary code; the compression control unit writes the generated compressed data into the memory according to the address of the compressed data, or the dictionary management unit writes the new dictionary code into the memory; the original data is read from the memory based on the address of the original data to start execution again; when the execution of the compression quick list is terminated, the compression control unit sends a calculation result notification to the compression agent of the database.
[0160] Reference Fig.14 According to an exemplary embodiment of the present disclosure, for the calculation process of the decompression quick list, on the database host side, the host CPU can control the decompression agent to send a fourth calculation request (decompression request) corresponding to the decompression quick list to the decompression control unit in the near storage computing device, so that the decompression control unit can start to execute the decompression quick list through the fourth calculation request, wherein the fourth calculation request can carry the address of the compressed data, the length of the compressed data, and the address of the decompressed data; in the process of the decompression control unit executing the decompression quick list, the decompression control unit can read the compressed data from the memory based on the address of the compressed data and the length of the compressed data; in the case where the decompression target of the compressed data has been completed, the execution of the decompression quick list is terminated; in the case where the decompression target of the compressed data has not been completed, the decompression control unit can call the dictionary management unit to find the dictionary code and call the second decoding unit to generate the decompressed original data with the length of the compressed data as the decompression target; the decompression control unit writes the generated decompressed original data into the memory based on the address of the decompressed data; and re-reads the compressed data from the memory based on the address of the compressed data to start execution; in the case where the execution of the decompression quick list is terminated, the decompression control unit sends a calculation result notification to the decompression agent of the database.
[0161] Fig.15A schematic diagram showing a comparison between a related technical solution and an exemplary embodiment of the present disclosure is shown.
[0162] Reference Fig.15 According to Example 1, in the CPU on the host side, events management, click / command line, exit and other operations can be performed. For the Redis database, all types of calculation processes, such as rehashing, list insertion, hashing, traversing compact lists, compression, list deletion, etc., are performed by the computing unit in the management system of the Redis database; therefore, the CPU on the host side directly accesses the memory (DRAM, Dynamic Random Access Memory) for data stream transmission. In Example 2 according to the present disclosure, the calculation types in the CPU on the host side are divided into memory-intensive calculations and non-memory-intensive calculations, and the memory-intensive calculations are offloaded to the near-memory computing device by setting a computing agent on the database host side and a computing core in the near-memory computing device. The computing agent sends a computing request and the computing core executes a computing process (for example, compression, decompression, rehashing, traversing a compact list, etc.), and the computing core in the near-memory computing device performs data transmission with the memory, so as to significantly reduce the data transmission between the CPU and the memory in the database, release the computing resources of the CPU in the database, save memory, and improve the database performance and execution efficiency. For non-memory-intensive calculations, such as hashing, list insertion, etc., since there is no need to frequently access the memory, there is no need to offload them, and all data exchanges can still be performed between the database host and the memory.
[0163] According to an exemplary embodiment of the present disclosure, the following table shows a comparison table of CPU memory access magnitudes of Example 1 and Example 2.
[0164]
[0165] As can be seen from the above table, for the calculation process of rehashing, the number of times the CPU accesses the memory in Example 1 is N (the number of entries) × M (the number of buckets), while the number in Example 2 is 0; for the calculation process of traversing the compact list, the number of times the CPU accesses the memory in Example 1 is N (the number of elements), while in Example 2, the CPU accesses the memory once only when inserting into the compact list after traversing the compact list; for the calculation process of compressing / decompressing the quick list, the number of times the CPU accesses the memory in Example 1 is N (the size of the node data), while the number in Example 2 is 0.
[0166] Fig.16 A block diagram showing a database computing management device according to an exemplary embodiment of the present disclosure.
[0167] Reference Fig.16The exemplary embodiment of the present disclosure further provides a database computing management device 1600. The database computing management device 1600 may include a request sending unit 1601 and a notification receiving unit 1602.
[0168] The request sending unit 1601 can send a corresponding computing request to the near memory computing device through a computing agent when the computing to be executed is a preset memory-intensive computing, wherein the memory-intensive computing is a computing in which the data transmission frequency between the processor and the memory is greater than a first preset threshold, and the computing agent is selected or determined for the memory-intensive computing.
[0169] The notification receiving unit 1602 may receive, through a computing agent, a notification of a computing result performed by a nearby computing device in response to a computing request.
[0170] According to an exemplary embodiment of the present disclosure, memory-intensive computing may include at least one of the following items: computing whose frequency of accessing memory is greater than a second preset threshold; computing whose simplicity of computing is less than a third preset threshold; computing whose simplicity of logical control is less than a fourth preset threshold.
[0171] According to an exemplary embodiment of the present disclosure, the memory-access intensive calculation further includes at least one of the following items: a re-hashing operation; a traversal operation; a compression operation or a decompression operation.
[0172] According to an exemplary embodiment of the present disclosure, the computing agent includes at least one of the following items: a first computing agent set for a re-hash operation; a second computing agent set for a traversal operation; and a third computing agent set for a compression / decompression operation.
[0173] According to an exemplary embodiment of the present disclosure, the above-mentioned computing management device 1600 may also include a computing execution unit (not shown in the figure) applied to a near-memory computing device, which can execute a corresponding computing process in response to a computing request through a computing core; wherein the computing core is set for memory-intensive computing, and the computing core includes at least one of the following items: a first computing core set for a re-hash operation; a second computing core set for a traversal operation; and a third computing core set for a compression / decompression operation.
[0174] According to an exemplary embodiment of the present disclosure, a computing execution unit can execute a corresponding computing process in response to a computing request through a process control unit and at least one computing unit in a computing core; wherein, at least one of the process control unit and at least one computing unit in the computing core is set for memory access intensive computing, and at least one of the process control unit and at least one computing unit in the computing core includes at least one of the following items: a re-hash control unit and a hash unit set for re-hash operations in a first computing core; a traversal control unit, a first decoding unit, and a comparison unit set for traversal operations in a second computing core; and a compression / decompression control unit, a second decoding unit, an encoding unit, and a dictionary management unit set for compression / decompression operations in a third computing core.
[0175] According to an exemplary embodiment of the present disclosure, a computing execution unit may receive a first computing request corresponding to a rehashing operation through a rehashing control unit, wherein the first computing request includes hash table information and new bucket size information; the rehashing control unit starts to execute rehashing based on the first computing request; the rehashing control unit sequentially reads the entry addresses of the buckets from the memory; when the entry address of the current bucket is empty, the rehashing is terminated; when the entry address of the current bucket is not empty, the rehashing control unit reads the first entry in the current bucket, and when the current bucket is empty, switches to read the entry address of the next bucket from the memory; the hash unit recalculates the hash key value according to the new bucket size information; the rehashing control unit reads the entry address of the new bucket; the rehashing control unit reattaches the entry to the new bucket, takes the next bucket of the current bucket as the new current bucket, and re-starts execution from reading the first entry in the current bucket; when the rehashing is terminated, the rehashing control unit sends a notification of the computing result to the computing agent of the database.
[0176] According to an exemplary embodiment of the present disclosure, a computing execution unit may receive a second computing request corresponding to a traversal of a compact list through a traversal control unit; the traversal control unit starts to execute the traversal of the compact list based on the second computing request; the traversal control unit reads the current element from the memory in sequence; when the address of the current element is empty, the traversal of the compact list is terminated; when the address of the current element is not empty, the first decoding unit performs a decoding calculation on the current element; the comparison unit compares the current element with the target element; based on determining that the current element is the target element, the traversal operation is terminated, and based on determining that the current element is not the target element, the traversal operation is restarted (for example, if yes, the traversal of the compact list is terminated, and if no, the current elements are read from the memory in sequence again to start execution); when the traversal of the compact list is terminated, the traversal control unit sends a notification of the calculation result to the calculation agent of the database.
[0177] According to an exemplary embodiment of the present disclosure, a calculation execution unit may receive a third calculation request corresponding to a compression quick list through a compression control unit, wherein the third calculation request carries the address of the original data, the length of the original data, and the address of the compressed data; the compression control unit starts to execute the compression quick list based on the third calculation request; the compression control unit reads the original data from the memory based on the address of the original data; when the compression target of the original data has been completed, the execution of the compression quick list is terminated; the compression control unit controls the encoding unit to generate at least one of the compressed data and the new dictionary code based on the dictionary management unit with the length of the original data as the compression target; the compression control unit writes at least one of the generated compressed data or the new dictionary code to the address of the compressed data, and restarts the compression operation based on the address of the original data (for example, restarts the execution by reading the original data from the memory based on the address of the original data); when the execution of the compression quick list is terminated, the compression control unit sends a notification of the calculation result to the calculation agent of the database.
[0178] According to an exemplary embodiment of the present disclosure, the computing execution unit may receive a fourth computing request corresponding to the decompression quick list through the decompression control unit, wherein the fourth computing request carries the address of the compressed data, the length of the compressed data, and the address of the decompressed data; the decompression control unit starts to execute the decompression quick list based on the fourth computing request; the decompression control unit reads the compressed data from the memory based on the address of the compressed data; when the decompression target of the compressed data has been completed, the execution of the decompression quick list is terminated; the decompression control unit controls the second decoding unit to generate decompressed data based on the dictionary management unit with the length of the compressed data as the decompression target; the decompression control unit writes the generated decompressed data to the address of the decompressed data, and restarts the decompression operation based on the address of the compressed data (for example, restarts the execution by reading the compressed data from the memory based on the address of the compressed data); when the execution of the decompression quick list is terminated, the decompression control unit sends a calculation result notification to the computing agent of the database.
[0179] It can be understood that each unit in the computing management device 1600 for the above-mentioned database is used to implement the computing management method for the above-mentioned database. The specific implementation process in the exemplary embodiment is roughly the same as the exemplary embodiment in the computing management method for the above-mentioned database, and will not be described repeatedly here. The computing management device 1600 for the database can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these units in the device may correspond to pure software code, or to a dedicated integrated circuit, or to a module that combines software and hardware. In addition, one or more functions implemented by these units in the device can also be uniformly executed by components in a physical entity device (for example, a processor, a client or a server, etc.).
[0180] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the computing management method of the above-mentioned database. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.
[0181] It should be noted that the computing management device of the database according to the exemplary embodiment of the present disclosure can completely rely on the operation of computer programs or instructions to realize the corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire device is called through a special software package (for example, lib library) to realize the corresponding function.
[0182] On the other hand, when the computing management device of the above-mentioned database is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.
[0183] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, a RAM or a flash memory is arranged within an integrated circuit microprocessor, etc. The computing device can read instructions stored in the storage device.
[0184] Another embodiment of the present disclosure relates to a system including at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above method embodiment.
[0185] According to the computing management method, device, storage medium and system of the database disclosed in the present invention, time-consuming memory-intensive calculations can be transferred from the database to a place close to the storage, such as to a computing memory such as a near memory computing (Processing Near Memory, PNM) device, which can significantly reduce the data movement between the CPU and memory in the database, release the CPU computing resources of the database, and thus fundamentally improve the efficiency of the database.
[0186] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.
Claims
1. A computing management method for managing a database, It is characterized in that The calculation management method comprises: Based on determining that the to-be-performed computation includes a preset memory-intensive computation, sending a computation request to a near-memory computing device through a computation agent, wherein a frequency of data transmission between a processor and a memory in the memory-intensive computation is greater than a first preset threshold, and the computation agent is selected to perform the memory-intensive computation; A notification is received, through the computing agent, regarding a computing result generated by the near-storage computing device based on the computing request.
2. The calculation management method according to claim 1, It is characterized in that The memory-intensive computation includes at least one of the following: The memory access frequency of the memory-intensive computing is greater than a second preset threshold; The computational simplicity of the memory-intensive computation is less than a third preset threshold; The logic control simplicity of the memory-intensive calculation is less than a fourth preset threshold.
3. The calculation management method according to claim 1, It is characterized in that The memory-intensive computation includes at least one of the following: Rehashing operations; Traversal operations corresponding to compact lists; Compress or decompress operations corresponding to quick lists.
4. The calculation management method according to claim 3, It is characterized in that The computing agent includes at least one of the following items: a first computing agent configured to perform the rehashing operation; a second computing agent configured to perform the traversal operation; A third computing agent is configured to perform the compression operation or the decompression operation.
5. The calculation management method according to claim 3, It is characterized in that The computing management method further includes: Executing a computing process based on the computing request by a computing core included in the near-storage computing device; The computing core is configured to perform the memory-intensive computing, and the computing core includes at least one of the following items: a first computing core configured to perform the rehashing operation; a second computing core configured to perform the traversal operation; A third computing core is configured to perform the compression operation or the decompression operation.
6. The calculation management method according to claim 5, It is characterized in that The performing a computing process based on the computing request by the computing core includes: executing the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core; Wherein, at least one of the process control unit and the at least one computing unit in the computing core is configured to perform the memory-intensive computation, and at least one of the process control unit and the at least one computing unit in the computing core includes at least one of the following items: a re-hashing control unit and a hashing unit in the first computing core configured to perform the re-hashing operation; a traversal control unit, a first decoding unit, and a comparison unit configured to perform the traversal operation in the second computing core; A compression control unit, a decompression control unit, a second decoding unit, an encoding unit and a dictionary management unit are configured in the third computing core to perform the compression operation or the decompression operation.
7. The calculation management method according to claim 6, It is characterized in that The step of executing the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core comprises: The re-hash control unit receives a first calculation request corresponding to the re-hash operation, wherein the first calculation request includes hash table information and new bucket size information; The re-hash control unit starts the re-hash operation based on the first calculation request; The re-hash control unit sequentially reads entry addresses of the storage buckets from the memory; End the rehashing based on the entry address of the current storage bucket being empty; Based on the entry address of the current storage bucket being not empty, the re-hash control unit reads the first entry in the current storage bucket, and based on the current storage bucket being empty, switches to read the entry address of the next storage bucket from the memory; The hash unit recalculates the hash key value according to the new bucket size information; The re-hash control unit reads an entry address of a new storage bucket; The re-hash control unit re-attaches the entry to the new storage bucket, uses a next storage bucket, and performs the re-hash operation based on the next bucket; Upon ending the re-hashing, the re-hashing control unit sends a notification regarding the calculation result to the calculation agent.
8. The calculation management method according to claim 6, It is characterized in that The step of executing the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core comprises: The traversal control unit receives a second calculation request corresponding to the traversal operation; The traversal control unit starts the traversal operation based on the second calculation request; The traversal control unit reads the current element from the memory in sequence; Based on the fact that the address of the current element is empty, the traversal operation is terminated; Based on the address of the current element being not empty, the first decoding unit performs a decoding calculation on the current element; The comparison unit compares the current element with the target element; Based on determining that the current element is the target element, ending the traversal operation, and based on determining that the current element is not the target element, restarting the traversal operation; Upon completion of the traversal operation, the traversal control unit sends a notification regarding the calculation result to the calculation agent.
9. The calculation management method according to claim 6, It is characterized in that The step of executing the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core comprises: The compression control unit in the near-storage computing device receives a third computing request corresponding to the compression operation, wherein the third computing request includes an address of original data, a length of the original data, and an address of compressed data; The compression control unit starts the compression operation based on the third calculation request; The compression control unit reads the original data from the memory based on the address of the original data; Ending the compression operation based on completing the compression target of the original data; The compression control unit controls the encoding unit to generate at least one of compressed data and a new dictionary code based on the dictionary management unit using the length of the original data as the compression target; The compression control unit writes at least one of the generated compressed data and the new dictionary code into the address of the compressed data, and restarts the compression operation based on the address of the original data; Upon ending the compression operation, the compression control unit sends a notification about the calculation result to the calculation agent.
10. The calculation management method according to claim 6, It is characterized in that The step of executing the computing process based on the computing request by a process control unit and at least one computing unit included in the computing core comprises: The decompression control unit receives a fourth calculation request corresponding to the decompression operation, wherein the fourth calculation request includes an address of compressed data, a length of the compressed data, and an address of decompressed data; The decompression control unit starts the decompression operation based on the fourth computing request; The decompression control unit reads the compressed data from the memory based on the address of the compressed data; Ending the decompression operation based on completing the decompression target of the compressed data; The decompression control unit controls the second decoding unit to generate decompressed data based on the dictionary management unit using the length of the compressed data as the decompression target; The decompression control unit writes the decompressed data to the address of the decompressed data and restarts the decompression operation based on the address of the compressed data; Based on ending the decompression operation, the decompression control unit sends a notification about the calculation result to the calculation agent.
11. A computing management device for managing a database, It is characterized in that The computing management device comprises: A request sending unit is configured to: send a computing request to a near-memory computing device through a computing agent based on determining that the computing to be executed includes a preset memory-intensive computing, wherein a frequency of data transmission between a processor and a memory in the memory-intensive computing is greater than a first preset threshold, and the computing agent is selected to execute the memory-intensive computing; The notification receiving unit is configured to: receive, through the computing agent, a notification about a computing result generated by the near-storage computing device based on the computing request.
12. A computer-readable storage medium storing instructions, in, When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the computing management method for managing a database as claimed in any one of claims 1 to 10.
13. A system comprising at least one computing device and at least one storage device storing instructions, in, When the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the computing management method for managing a database according to any one of claims 1 to 10.
