Database distributed memory management method and device, equipment and storage medium

Abstract

The invention discloses a database distributed memory management method and device, equipment and a storage medium, and relates to the field of databases, and the method comprises the following steps: in a target database, constructing a distributed memory pooling framework by taking a database instance as a node, and under the distributed memory pooling framework, creating a mapping table by utilizing a memory mapping technology; accessing the shared cache page based on the mapping table among the nodes, and dynamically adjusting the distribution of the private cache page through a preset adaptive load monitoring algorithm and a preset load weight determination formula; and when a data write-in request of the node is obtained, controlling the request flow based on a token bucket algorithm, and executing a write operation on to-be-modified data located in the shared cache page in the node by using a timestamp-based optimistic lock mechanism. Therefore, the management efficiency of the distributed memory in the database can be improved.

Description

Technical Field

[0001] The present invention relates to the field of databases, and particularly to a method, device, equipment and storage medium for distributed memory management of databases. Background Art

[0002] In today's digital age, the amount of data has grown explosively, and database systems are facing unprecedented challenges. Existing databases have many deficiencies in memory management. In distributed databases, although multi-node collaboration has been achieved, there is a lack of an effective coordination mechanism for memory management. The memory utilization rate of each node is low. At the same time, when there is a data write request at a node, due to the lack of a suitable traffic control strategy, a large number of concurrent requests often cause network congestion, seriously affecting the overall performance and stability of the system.

[0003] In summary, how to improve the efficiency of distributed memory management in databases is a technical problem that needs to be solved urgently at present. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for distributed memory management of databases, which can improve the efficiency of distributed memory management in databases. The specific solutions are as follows:

[0005] In a first aspect, the present application provides a method for distributed memory management of a database, including:

[0006] In a target database, a distributed memory pooling framework is constructed with database instances as nodes, and under the distributed memory pooling framework, a mapping table is created using memory mapping technology; wherein, the mapping table is a table representing the correspondence between the virtual address and the physical address of a shared cache page;

[0007] Among the nodes, based on the mapping table, access the shared cache page, and dynamically adjust the allocation of private cache pages through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache page is a cache page in the shared memory of each node, and the private cache page is a cache page in the private memory corresponding to the node, and different nodes have different private memories;

[0008] When a data write request of the node is obtained, control the request traffic based on the token bucket algorithm, and perform a write operation on the data to be modified located in the shared cache page of the node using an optimistic lock mechanism based on a timestamp; wherein, the request traffic is the number of requests initiated by each node per unit time for writing the data to be modified.

[0009] Optionally, accessing the shared cache page based on the mapping table and dynamically adjusting the allocation of the private cache page through a preset adaptive load monitoring algorithm and a preset load weight determination formula includes:

[0010] Determine the virtual address of the shared cache page, and determine the physical address of the shared cache page based on the virtual address and the mapping table, so as to access the shared cache page using the physical address;

[0011] Use a preset adaptive load monitoring algorithm to obtain the CPU usage rate, memory utilization rate, and disk I / O rate of each node to determine monitoring information;

[0012] Based on the monitoring information, the preset adaptive load monitoring algorithm, and the preset load weight determination formula, determine the current load situation of each node, and use the load situation to dynamically adjust the allocation of the private cache page.

[0013] Optionally, controlling the request traffic based on the token bucket algorithm includes:

[0014] Generate a token set at a preset fixed rate based on the token bucket in the token bucket algorithm;

[0015] Send the target token in the token set for the data write request, so that the data write request performs a preset data write operation based on the target token;

[0016] When there is no token in the token bucket, transfer the current data write request to a preset request waiting queue.

[0017] Optionally, performing a write operation on the data to be modified located in the shared cache page of the node using an optimistic lock mechanism based on a timestamp includes:

[0018] Use an optimistic lock mechanism based on a timestamp to obtain a first timestamp representing the data save time corresponding to the data to be modified before the write operation is executed. When the write operation is executed, use the optimistic lock mechanism based on the timestamp to obtain a second timestamp representing the current data save time of the data to be modified;

[0019] Compare the first timestamp and the second timestamp, and obtain a comparison result;

[0020] If the comparison result indicates that the first timestamp and the second timestamp are the same, allow a write operation to be performed on the data to be modified located in the shared cache page of the node, and modify the second timestamp to the timestamp corresponding after the write operation is completed.

[0021] Optionally, the database distributed memory management method further includes:

[0022] When transferring data between each of the nodes, determine the type of the preset data transfer protocol;

[0023] If the preset data transfer protocol is the Transmission Control Protocol, adjust the window size of the data to be transferred based on the cubic algorithm and the current network congestion situation, and transfer the data between each of the nodes by using the adjusted window;

[0024] If the preset data transfer protocol is the Remote Direct Memory Access protocol, register the local memory as remote memory based on the memory registration mechanism of the Remote Direct Memory Access protocol, so as to transfer data between each of the nodes by using the application programming interface of the Remote Direct Memory Access protocol.

[0025] Optionally, the method for distributed memory management of a database further includes:

[0026] Maintain a plurality of free memory block linked lists on each of the nodes based on the buddy system algorithm;

[0027] When receiving a memory allocation request, allocate a memory block from the plurality of free memory block linked lists;

[0028] Use a data monitoring tool to monitor the memory usage of the memory blocks of each of the nodes and obtain a monitoring result;

[0029] When the monitoring result indicates that there is a node with a memory utilization rate greater than a preset utilization rate threshold, use the incremental data migration algorithm to migrate the data in the memory blocks of the node with a memory utilization rate greater than the preset utilization rate threshold.

[0030] Optionally, the method for distributed memory management of a database further includes:

[0031] When modifying the private cache page in the target node, determine the network communication standard;

[0032] Broadcast a consistency message by the target node to other nodes except the target node based on the network communication standard and a preset cache coherence protocol, so that after receiving the consistency message, the other nodes update the status of their respective private cache pages based on the preset cache coherence protocol.

[0033] In a second aspect, the present application provides a device for distributed memory management of a database,

[0034] including:

[0035] A mapping table determination module, configured to construct a distributed memory pooling framework with database instances as nodes in a target database, and create a mapping table by using a memory mapping technique under the distributed memory pooling framework; wherein, the mapping table is a table representing the correspondence between the virtual address and the physical address of a shared cache page.

[0036] A page allocation module, configured to access shared cache pages among the nodes based on the mapping table, and dynamically adjust the allocation of private cache pages through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache pages are cache pages in the shared memory of each node, and the private cache pages are cache pages in the private memory corresponding to the node, and different nodes have different private memories.

[0037] A data writing module, configured to, when a data writing request of the node is obtained, control the request traffic based on a token bucket algorithm, and perform a write operation on the data to be modified located in the shared cache page of the node by using an optimistic lock mechanism based on a timestamp; wherein, the request traffic is the number of requests initiated by each node per unit time for writing the data to be modified.

[0038] In a third aspect, the present application provides an electronic device, including:

[0039] A memory, configured to store a computer program;

[0040] A processor, configured to execute the computer program to implement the foregoing database distributed memory management method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program; wherein, when the computer program is executed by a processor, the foregoing database distributed memory management method is implemented.

[0042] In this application, in the target database, a distributed memory pooling framework is constructed with database instances as nodes. Under the distributed memory pooling framework, a mapping table is created using memory mapping technology; wherein, the mapping table is a table representing the correspondence between the virtual address and the physical address of the shared cache page; between each of the nodes, the shared cache page is accessed based on the mapping table, and the allocation of private cache pages is dynamically adjusted through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache page is the cache page in the shared memory of each node, and the private cache page is the cache page in the private memory corresponding to the node, and different nodes have different private memories; when a data write request of the node is obtained, the request traffic is controlled based on the token bucket algorithm, and a write operation is performed on the data to be modified located in the shared cache page of the node using an optimistic lock mechanism based on timestamps; wherein, the request traffic is the number of requests initiated by each node per unit time for writing the data to be modified. As can be seen from the above, in the target database, the database instance is used as a node to construct a distributed memory pooling framework. In this distributed memory pooling framework, the memory mapping technology is used to create a mapping table. Here, the mapping table is a table used to represent the correspondence between the virtual address and the physical address of the shared cache page. Between each node, the shared cache page is accessed according to the created mapping table. At the same time, with the help of a preset adaptive load monitoring algorithm and a preset load weight determination formula, the allocation of private cache pages is dynamically adjusted. Among them, the shared cache page is the cache page in the shared memory of each node, and the private cache page is the cache page in the private memory corresponding to each node, and the private memories of different nodes are different. When a data write request sent by the node is received, the request traffic is controlled based on the token bucket algorithm. And, an optimistic lock mechanism based on timestamps is used to perform a write operation on the data to be modified in the shared cache page of the node. In this way, this application can achieve efficient management and flexible allocation of memory resources, thereby improving the overall performance of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0044] Figure 1 It is a flowchart of a database distributed memory management method disclosed in this application;

[0045] Figure 2Flowchart of a specific database distributed memory management method disclosed in this application;

[0046] Figure 3 Schematic diagram of the position of a database distributed memory management method in the database architecture disclosed in this application;

[0047] Figure 4 Schematic diagram of the structure of a database distributed memory management device disclosed in this application;

[0048] Figure 5 Structural diagram of an electronic device disclosed in this application. Specific implementation manners

[0049] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0050] Currently, there are many deficiencies in database memory management. In a distributed database, although multi-node collaboration is achieved, there is a lack of an effective coordination mechanism for memory management. The memory utilization rate of each node is low. At the same time, when there is a data write request at a node, due to the lack of a suitable traffic control strategy, a large number of concurrent requests often cause network congestion, seriously affecting the overall performance and stability of the system. For this reason, this application provides a database distributed memory management method, device, device, and storage medium, which can improve the efficiency of distributed memory management in the database.

[0051] See Figure 1 As shown, the embodiments of the present invention disclose a database distributed memory management method, including:

[0052] Step S11: In the target database, construct a distributed memory pooling framework with database instances as nodes. Under the distributed memory pooling framework, create a mapping table using memory mapping technology; wherein, the mapping table is a table representing the correspondence between the virtual address and the physical address of the shared cache page.

[0053] In this embodiment, in the target database, a distributed memory pooling framework is constructed with database instances as nodes. Under this distributed memory pooling framework, a mapping table is created using memory mapping technology. Here, the mapping table is specifically a table that can represent the correspondence between the virtual address and the physical address of the shared cache page. By creating such a mapping table, during the subsequent access to the shared cache page, the conversion from the virtual address to the physical address can be more accurately achieved.

[0054] In addition, for each node in the target database, a number of free memory block linked lists are maintained on each node based on the buddy system algorithm. Among them, the buddy system algorithm divides large free memory blocks according to the power of 2, forming free memory block linked lists of different sizes. Therefore, when a memory allocation request arrives, the free block can be quickly found and allocated from the linked list of the appropriate size according to the requested memory size. If a free block of the appropriate size cannot be found in the linked list of a certain size, a larger free block is obtained from the linked list and split to meet the memory allocation requirements.

[0055] When a memory allocation request is received, a memory block will be allocated from several free memory block linked lists maintained according to the above-mentioned buddy system algorithm. During the allocation process, factors such as the size of the free memory block and the usage of the linked list will be comprehensively considered to ensure the rationality of memory block allocation.

[0056] To better manage memory resources, a data monitoring tool is used to monitor the memory usage of memory blocks on each node and obtain the monitoring results. Here, the data monitoring tool can collect real-time data related to the memory usage of each node, such as memory utilization rate, memory read and write speed, etc.

[0057] When the obtained monitoring results indicate that there is a node with a memory utilization rate greater than the preset utilization threshold, it means that the memory resources of this node are already in a relatively tense state. To optimize the overall memory usage and improve the performance and stability of the system, the incremental data migration algorithm is used to migrate the data in the memory blocks of the node with a memory utilization rate greater than the preset utilization threshold. The incremental data migration algorithm will first migrate the data blocks with frequent modifications, which can reduce the amount of data migration and improve the migration efficiency. In this way, migrating some data from the node with too high memory utilization rate to other nodes with more free memory blocks can optimize the allocation of memory resources among nodes.

[0058] Step S12: Access the shared cache page based on the mapping table among the nodes, and dynamically adjust the allocation of the private cache page through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache page is the cache page in the shared memory of each node, and the private cache page is the cache page in the private memory corresponding to the node. Different nodes have different private memories.

[0059] In this embodiment, among nodes, when a need to access a shared cache page is obtained, first determine the virtual address of the shared cache page, and based on the virtual address and the mapping table, determine the physical address of the shared cache page, so as to access the shared cache page using the physical address. It should be noted that in a distributed memory pooling framework, a shared cache page is a data storage area that can be jointly accessed by multiple nodes. When a certain node needs to access a shared cache page, first determine the virtual address of the page, and this process can be completed through the address management mechanism inside the target database. Further, according to the mapping table, convert the determined virtual address into a physical address. Once the physical address is determined, the node can directly access the shared cache page to obtain the required data.

[0060] For the allocation of private cache pages, first use a preset adaptive load monitoring algorithm to obtain the CPU usage rate, memory utilization rate, and disk I / O rate of each of the nodes to determine monitoring information. Among them, the preset adaptive load monitoring algorithm is an algorithm that can collect the system performance indicators of each node in real time and accurately.

[0061] Further, based on the monitoring information, the preset adaptive load monitoring algorithm, and a preset load weight determination formula, determine the current load conditions of each of the nodes, and use the load conditions to dynamically adjust the allocation of private cache pages. According to the load conditions of each node, the allocation of private cache pages can be dynamically adjusted. If the load of a certain node is too high, then migrate some of the private cache pages of that node to a node with a lower load to balance the loads of the nodes.

[0062] During the process of modifying the private cache page at the target node, determine the network communication standard, and broadcast a consistency message from the target node to other nodes except the target node based on the network communication standard and the preset cache coherence protocol, so that after receiving the consistency message, the other nodes update the status of their respective private cache pages based on the preset cache coherence protocol. Among them, the preset cache coherence protocol is an important mechanism to ensure the consistency of the private cache page status of each node. When the target node modifies the private cache page, it will broadcast a consistency message to other nodes according to the network communication standard, notifying other nodes that the status of this page has changed. After receiving the message, other nodes will update the status of their respective private cache pages according to the rules of the preset cache coherence protocol to ensure that the private cache pages of each node always remain consistent. For example, when using the MESI protocol (i.e., Modified Exclusive Shared Invalid, a cache coherence protocol based on Invalidate), after the target node modifies the cache page, it will broadcast a message to inform other nodes that the status of this page has become "modified", and after receiving the message, other nodes will update the status of the local page accordingly.

[0063] Step S13: When a data write request of the node is obtained, control the request traffic based on the token bucket algorithm, and perform a write operation on the data to be modified located in the shared cache page of the node by using an optimistic lock mechanism based on timestamps; wherein, the request traffic is the number of requests initiated by each node per unit time for writing the data to be modified.

[0064] In this embodiment, when a data write request of the node is obtained, a token set is generated at a preset fixed rate based on the token bucket in the token bucket algorithm. Among them, the token bucket algorithm is a commonly used traffic control algorithm. In this algorithm, the token bucket continuously generates tokens at a preset fixed rate.

[0065] Furthermore, send the target token in the token set for the data write request, so that the data write request performs a preset data write operation based on the target token. Specifically, when a certain node initiates a data write request, the system takes out a token from the token bucket and assigns it to the request. Only the requests that obtain the token are allowed to perform the preset data write operation.

[0066] In a specific embodiment, when there is no token in the token bucket, the current data write request is transmitted to a preset request waiting queue. That is to say, if all the tokens in the token bucket have been taken, it means that the processing capacity of the system has reached the upper limit at this time. To ensure the stability of the system and the consistency of data, new data write requests will be placed in the preset request waiting queue to wait. The requests in the waiting queue will follow the first-in, first-out principle. When new tokens are generated again in the token bucket, the requests will be taken out of the queue in turn and tokens will be allocated to them so that they can perform data write operations.

[0067] During the execution of the write operation, first, the first timestamp representing the data storage time of the data to be modified before the write operation is executed is obtained by using the optimistic lock mechanism based on the timestamp. When the write operation is being executed, the second timestamp representing the current data storage time of the data to be modified is obtained by using the optimistic lock mechanism based on the timestamp. Then, the first timestamp and the second timestamp are compared, and a comparison result is obtained. By comparing these two timestamps, it can be judged whether the data has been modified by other nodes during the execution of the write operation. If the first timestamp and the second timestamp are the same, it means that the data has not been modified by other nodes during the execution of the write operation, and the current write operation is safe; if the two timestamps are different, it means that the data has been modified by other nodes, and the current write operation may cause data inconsistency.

[0068] If the comparison result shows that the first timestamp and the second timestamp are the same, the write operation on the data to be modified located in the shared cache page of the node is allowed to be executed, and the second timestamp is modified to the timestamp corresponding after the write operation is completed. When it is judged that the write operation is safe, the write operation will be allowed to continue to be executed. After the write operation is completed, the second timestamp will be updated and modified to the time after the write operation is completed. In this way, when other nodes perform write operations on this data subsequently, they can judge whether the data has been modified according to the new timestamp, so as to ensure data consistency.

[0069] As can be seen from the above, in the target database, the database instance is used as a node to construct a distributed memory pooling framework. In this distributed memory pooling framework, the memory mapping technology is used to create a mapping table. Here, the mapping table is a table used to represent the correspondence between the virtual address and the physical address of the shared cache page. Among the various nodes, the shared cache page is accessed according to the created mapping table. At the same time, with the help of a preset adaptive load monitoring algorithm and a preset load weight determination formula, the allocation of private cache pages is dynamically adjusted. Among them, the shared cache page is the cache page in the shared memory of each node, while the private cache page is the cache page in the private memory corresponding to each node, and the private memories of different nodes are different. When a data write request sent by a node is received, the request traffic is controlled based on the token bucket algorithm. And, using an optimistic lock mechanism based on timestamps, a write operation is performed on the data to be modified in the shared cache page in the node. In this way, the present application can achieve efficient management and flexible allocation of memory resources, thereby improving the overall performance of the database.

[0070] See Figure 2 As shown, in order to achieve efficient data transfer between nodes in the target database, an embodiment of the present application further provides a specific database distributed memory management method, including:

[0071] Step S21, when transferring data between each of the nodes, determine the type of a preset data transfer protocol.

[0072] In this embodiment, when transferring data between nodes, it is first necessary to determine the type of the preset data transfer protocol. Common preset data transfer protocols mainly include the Transmission Control Protocol (i.e., TCP, Transmission Control Protocol) and the Remote Direct Memory Access (i.e., RDMA, Remote Direct Memory Access) protocol.

[0073] It can be understood that in this embodiment, which data transfer protocol to use can be determined according to factors such as specific business requirements, network environment, and data characteristics. For example, when performing operations with high requirements for data accuracy such as data backup and synchronization, the Transmission Control Protocol may be selected, while in scenarios with strict requirements for data transfer speed such as real-time transmission of large-scale data and high-speed computing, the Remote Direct Memory Access protocol is more preferred.

[0074] Step S22, if the preset data transfer protocol is the Transmission Control Protocol, adjust the window size of the data to be transferred based on the cubic algorithm and the current network congestion situation, and transfer the data between each of the nodes using the adjusted window.

[0075] In this embodiment, if the preset data transmission protocol is the Transmission Control Protocol (TCP), the data sending window size is adjusted based on the cubic algorithm (i.e., the CUBIC algorithm, a congestion control algorithm) and the current network congestion situation, and the adjusted window is used to transfer data between nodes. Among them, the cubic algorithm adapts to the network congestion situation by dynamically adjusting the data sending window size. When the network congestion level is low, the data sending window can be appropriately increased, and when the network congestion level is high, the data sending window will be correspondingly decreased to avoid too much data flooding into the network and further exacerbating congestion.

[0076] Using the adjusted window to transfer data between nodes can ensure reliable data transmission while improving the data transmission efficiency as much as possible. By continuously adjusting the window size according to the network congestion situation, the Transmission Control Protocol can adaptively optimize the data transmission process and avoid problems such as increased data transmission delay or data loss caused by network congestion.

[0077] Step S23: If the preset data transmission protocol is the Remote Direct Memory Access (RDMA) protocol, register the local memory as remote memory based on the memory registration mechanism of the Remote Direct Memory Access protocol, so as to transfer data between the nodes using the application programming interface of the Remote Direct Memory Access protocol.

[0078] In this embodiment, if the preset data transmission protocol is the Remote Direct Memory Access protocol, the local memory is registered as remote memory based on the memory registration mechanism of this protocol, so as to transfer data between nodes using its application programming interface.

[0079] It should be noted that the memory registration mechanism of the Remote Direct Memory Access protocol allows the local memory area to be registered as memory that can be directly accessed by remote nodes. Before data transmission, the node will register the local memory area where the data to be transmitted is located through this mechanism, so that the remote node can directly access this memory area without copying data through the operating system kernel of the local node, thereby reducing the data transmission overhead.

[0080] At the software level, use the application programming interface provided by the Remote Direct Memory Access protocol, such as Verbs API (a set of the most basic software interfaces for using RDMA services), to enable the application program to directly access the memory of remote nodes. For example, the application program can directly write data from the local memory to the memory of the remote node or read data from the memory of the remote node to the local memory through these interfaces without going through the traditional network transmission and data copying process.

[0081] As can be seen from the above, by selecting an appropriate data transmission protocol according to different business requirements and network environments and taking corresponding operations to optimize the data transmission process, that is to say, during the data transmission process, if the Transmission Control Protocol is adopted, the cubic algorithm will be used in combination with the real-time network congestion situation to dynamically adjust the window size of the data to be transmitted, so as to ensure that the data is transmitted as efficiently as possible on the basis of reliable transmission. When the Remote Direct Memory Access protocol is selected, its memory registration mechanism is used to register the local memory as remote memory, and with the help of a dedicated application programming interface, the application program can directly access the memory of the remote node, greatly reducing the overhead of data transmission.

[0082] See Figure 3 as shown Figure 3 in the figure is the location diagram of distributed memory management in the database architecture, that is, the technical solution of this application is reflected in Figure 3 the "Distributed Memory Service" in. Specifically, as shown in the figure, the client can perform read and write operations on the primary node and only read operations on the standby node. There are database primary instances in both the primary node and the standby node, which cover the distributed memory service and the distributed storage service. They cooperate with the shared memory and the distributed storage service instances, and data is efficiently transmitted between the primary node and the standby node through the RDMA technology. At the same time, both the primary node and the standby node can read and write to the shared disk. Therefore, the technical solution of this application realizes the unified management and allocation of memory resources through interaction with other components in the figure. Through this architecture, the performance, reliability, and data processing ability of the database system can be improved.

[0083] Correspondingly, see Figure 4 as shown, the embodiment of this application provides a database distributed memory management device, including:

[0084] A mapping table determination module 11, configured to build a distributed memory pooling framework with database instances as nodes in the target database, and create a mapping table using memory mapping technology under the distributed memory pooling framework; wherein, the mapping table is a table representing the correspondence between the virtual address and the physical address of the shared cache page;

[0085] A page allocation module 12, configured to access the shared cache page based on the mapping table between the nodes, and dynamically adjust the allocation of private cache pages through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache page is a cache page in the shared memory of each node, and the private cache page is a cache page in the private memory corresponding to the node, and different nodes have different private memories;

[0086] A data writing module 13, which is used to, when receiving the data writing request of the node, control the request traffic based on the token bucket algorithm, and perform a write operation on the data to be modified located in the shared cache page of the node by using an optimistic lock mechanism based on timestamps; wherein, the request traffic is the number of requests initiated by each node for writing the data to be modified per unit time.

[0087] As can be seen from the above, in the target database, the database instance is used as a node to construct a distributed memory pooling framework. In this distributed memory pooling framework, the memory mapping technology is used to create a mapping table. Here, the mapping table is a table used to represent the correspondence between the virtual address and the physical address of the shared cache page. Among the various nodes, the shared cache page is accessed according to the created mapping table. At the same time, with the help of a preset adaptive load monitoring algorithm and a preset load weight determination formula, the allocation of the private cache page is dynamically adjusted. Among them, the shared cache page is the cache page in the shared memory of each node, and the private cache page is the cache page in the private memory corresponding to each node, and the private memories of different nodes are different. When receiving the data writing request sent by the node, the request traffic is controlled based on the token bucket algorithm. And, using an optimistic lock mechanism based on timestamps, a write operation is performed on the data to be modified in the shared cache page of the node. In this way, the present application can achieve efficient management and flexible allocation of memory resources, thereby improving the overall performance of the database.

[0088] In some specific embodiments, the page allocation module 12 specifically includes:

[0089] A page memory access unit, which is used to determine the virtual address of the shared cache page, and determine the physical address of the shared cache page based on the virtual address and the mapping table, so as to access the shared cache page by using the physical address;

[0090] An information monitoring unit, which is used to obtain the CPU usage rate, memory utilization rate and disk I / O rate of each node by using a preset adaptive load monitoring algorithm, so as to determine the monitoring information;

[0091] A page allocation unit, which is used to determine the current load situation of each node based on the monitoring information, the preset adaptive load monitoring algorithm and the preset load weight determination formula, and dynamically adjust the allocation of the private cache page by using the load situation.

[0092] In some specific embodiments, the data writing module 13 specifically includes:

[0093] A token generation unit, which is used to generate a token set at a preset fixed rate based on the token bucket in the token bucket algorithm;

[0094] A token sending unit, configured to send a target token in the token set for the data writing request, so that the data writing request performs a preset data writing operation based on the target token;

[0095] A request transmission unit, configured to transmit the current data writing request to a preset request waiting queue when there is no token in the token bucket.

[0096] In some specific embodiments, the data writing module 13 specifically includes:

[0097] A timestamp acquisition unit, configured to acquire a first timestamp representing the data storage time of the data to be modified before the write operation is executed by using an optimistic lock mechanism based on timestamps, and acquire a second timestamp representing the current data storage time of the data to be modified when the write operation is executed by using the optimistic lock mechanism based on timestamps;

[0098] A timestamp comparison unit, configured to compare the first timestamp and the second timestamp and obtain a comparison result;

[0099] A data modification unit, configured to, if the comparison result indicates that the first timestamp and the second timestamp are the same, allow a write operation to be performed on the data to be modified located in the shared cache page of the node, and modify the second timestamp to a timestamp corresponding after the write operation is completed.

[0100] In some specific embodiments, the database distributed memory management device specifically further includes:

[0101] A protocol determination unit, configured to determine the type of a preset data transmission protocol when data is transmitted between the nodes;

[0102] A first data transmission unit, configured to, if the preset data transmission protocol is the Transmission Control Protocol, adjust the window size of the data to be transmitted based on the cubic algorithm and the current network congestion situation, and transmit the data between the nodes by using the adjusted window;

[0103] A second data transmission unit, configured to, if the preset data transmission protocol is the Remote Direct Memory Access protocol, register the local memory as a remote memory based on the memory registration mechanism of the Remote Direct Memory Access protocol, so as to transmit the data between the nodes by using the application programming interface of the Remote Direct Memory Access protocol.

[0104] In some specific embodiments, the database distributed memory management device specifically further includes:

[0105] A linked list maintenance unit, configured to maintain a plurality of free memory block linked lists on each of the nodes based on the buddy system algorithm;

[0106] A memory allocation unit, configured to allocate a memory block from several linked lists of free memory blocks when receiving a memory allocation request;

[0107] A memory status monitoring unit, configured to monitor the memory usage of the memory blocks of each node by using a data monitoring tool and obtain a monitoring result;

[0108] A data migration unit, configured to, when the monitoring result indicates that there is a node with a memory utilization rate greater than a preset utilization rate threshold, migrate the data in the memory blocks of the node with a memory utilization rate greater than the preset utilization rate threshold by using an incremental data migration algorithm.

[0109] In some specific embodiments, the database distributed memory management device further specifically includes:

[0110] A standard determination unit, configured to determine a network communication standard during the process of a target node modifying the private cache page;

[0111] A message broadcasting unit, configured to broadcast a consistency message to other nodes except the target node by the target node based on the network communication standard and a preset cache consistency protocol, so that after receiving the consistency message, the other nodes update the states of their respective private cache pages based on the preset cache consistency protocol.

[0112] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 5 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation to the usage scope of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the database distributed memory management method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0113] In this embodiment, the power supply 23 is used to provide a working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0114] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a magnetic disk, an optical disc, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0115] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the database distributed memory management method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0116] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the database distributed memory management method disclosed above. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated herein.

[0117] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0118] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0119] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0120] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0121] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this text to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A database distributed memory management method, characterized in that: include: In the target database, a distributed memory pooling framework is constructed with database instances as nodes, and a mapping table is created using memory mapping technology under the distributed memory pooling framework; wherein the mapping table is a table representing the correspondence between the virtual address and the physical address of the shared cache page; Between the nodes, the shared cache page is accessed based on the mapping table, and the allocation of the private cache page is dynamically adjusted through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache page is a cache page in the shared memory of each node, and the private cache page is a cache page in the private memory corresponding to the node, and different nodes have different private memories; When a data write request is obtained from the node, the request flow is controlled based on the token bucket algorithm, and a timestamp-based optimistic locking mechanism is used to perform a write operation on the data to be modified located in the shared cache page in the node; wherein the request flow is the number of requests for write operations on the data to be modified initiated by each node within a unit time.

2. The database distributed memory management method according to claim 1, characterized in that: The accessing of the shared cache page based on the mapping table and dynamically adjusting the allocation of the private cache page by a preset adaptive load monitoring algorithm and a preset load weight determination formula include: Determine a virtual address of a shared cache page, and determine a physical address of the shared cache page based on the virtual address and the mapping table, so as to access the shared cache page using the physical address; Using a preset adaptive load monitoring algorithm to obtain the CPU usage, memory utilization and disk I / O rate of each of the nodes to determine monitoring information; The current load condition of each of the nodes is determined based on the monitoring information, the preset adaptive load monitoring algorithm, and a preset load weight determination formula, and the allocation of private cache pages is dynamically adjusted using the load condition.

3. The database distributed memory management method according to claim 1, characterized in that: The token bucket algorithm-based request flow control includes: Generate a token set at a preset fixed rate based on a token bucket in a token bucket algorithm; Sending a target token in the token set for the data write request, so that the data write request performs a preset data write operation based on the target token; When there is no token in the token bucket, the current data write request is transmitted to a preset request waiting queue.

4. The database distributed memory management method according to claim 1, characterized in that: The using of the timestamp-based optimistic locking mechanism to perform a write operation on the to-be-modified data located in the shared cache page in the node includes: Using an optimistic locking mechanism based on timestamps to obtain a first timestamp representing the data storage time corresponding to the data to be modified before the write operation is performed, and when the write operation is performed, using the optimistic locking mechanism based on timestamps to obtain a second timestamp representing the data storage time of the data to be modified; Comparing the first timestamp and the second timestamp, and obtaining a comparison result; If the comparison result shows that the first timestamp and the second timestamp are the same, the write operation is allowed to be performed on the data to be modified in the shared cache page in the node, and the second timestamp is modified to a timestamp corresponding to the completion of the write operation.

5. The database distributed memory management method according to claim 1, characterized in that: Also includes: When transferring data between the nodes, determining the type of the preset data transmission protocol; If the preset data transmission protocol is a transmission control protocol, adjusting the window size of the data to be transmitted based on the cubic algorithm and the current network congestion situation, and transmitting the data between the nodes using the adjusted window; If the preset data transmission protocol is the remote direct memory access protocol, the local memory is registered as the remote memory based on the memory registration mechanism of the remote direct memory access protocol, so as to transfer data between the nodes using the application programming interface of the remote direct memory access protocol.

6. The database distributed memory management method according to claim 1, characterized in that: Also includes: Maintaining a number of free memory block linked lists on each of the nodes based on a buddy system algorithm; When a memory allocation request is received, a memory block is allocated from the plurality of free memory block linked lists; Using a data monitoring tool to monitor the memory usage of the memory blocks of each of the nodes and obtain monitoring results; When the monitoring result indicates that there is a node whose memory utilization is greater than a preset utilization threshold, an incremental data migration algorithm is used to migrate data in the memory block of the node whose memory utilization is greater than the preset utilization threshold.

7. The database distributed memory management method according to any one of claims 1 to 6, characterized in that: Also includes: During the process of modifying the private cache page at the target node, determining a network communication standard; The target node broadcasts a consistency message to other nodes except the target node based on the network communication standard and the preset cache consistency protocol, so that after receiving the consistency message, the other nodes update the status of their respective private cache pages based on the preset cache consistency protocol.

8. A database distributed memory management device, characterized in that: include: A mapping table determination module is used to construct a distributed memory pooling framework in a target database with database instances as nodes, and to create a mapping table using memory mapping technology under the distributed memory pooling framework; wherein the mapping table is a table representing the correspondence between the virtual address and the physical address of a shared cache page; A page allocation module, used to access shared cache pages between the nodes based on the mapping table, and dynamically adjust the allocation of private cache pages through a preset adaptive load monitoring algorithm and a preset load weight determination formula; the shared cache pages are cache pages in the shared memory of each node, the private cache pages are cache pages in the private memory corresponding to the node, and different nodes have different private memories; A data writing module is used to control the request flow based on the token bucket algorithm when a data writing request of the node is obtained, and to perform a write operation on the data to be modified located in the shared cache page in the node using an optimistic locking mechanism based on a timestamp; wherein the request flow is the number of requests initiated by each node to perform write operations on the data to be modified within a unit time.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the database distributed memory management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the database distributed memory management method as described in any one of claims 1 to 7 is implemented.

Images