Collaborative memory management in database management system
By introducing a dynamic memory manager in the database management system, adjusting the memory allocation of table data and temporary data, the problem of static and unused memory allocation in traditional systems is solved, performance and cost efficiency are improved, and efficient memory utilization is achieved.
Patent Information
- Application Number
- CN202410574793.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-12
- Filing Date
- 2024-05-10
- Publication Date
- 2024-11-12
AI Technical Summary
In traditional database management systems, memory allocation is static and separate, resulting in a large amount of memory not being used when performing small queries or no queries, and configuring the buffer pool size is time-consuming and error-prone, which may cause query failure or overflow.
The dynamic memory manager is introduced to ensure the effective utilization of the storage buffer by adjusting the memory allocation between table data and temporary data during querying or querying different parts one by one, and dynamically modify the allocation to meet query needs.
Improves the performance and cost efficiency of the database management system, avoids unused memory and complex configuration problems, and ensures that all main memory can be effectively utilized when the memory limit is reached.
Smart Images

Figure CN118939646A_ABST
Abstract
Description
Background Art
[0001] The present disclosure relates generally to database software systems, and in particular to collaborative memory management in database management systems.
[0002] Main memory requirements are a significant factor in the cost of operating a database management system (DBMS). In systems running analytical or mixed analytical and transactional workloads, while many individual consumers require memory, the majority of memory capacity is required by two types of consumers. First, table data is loaded and held in memory to read and update it. Second, running complex analytical queries requires memory for temporary data.
[0003] Traditionally, disk-based systems manage memory for table data separately from all other memory. They load table data from storage into memory via a buffer manager, which manages a pool of memory to cache accessed table data. The size of this pool is user-configured, and memory allocated to it cannot be used for other consumers in the system like temporary data.
[0004] However, the traditional paradigm for managing memory leads to several problems. Since the buffer pool size needs to be configured so that enough storage capacity is left to hold the peak amount of temporary data created by any query in the workload, much of the memory capacity left for temporary data will remain unused during times when smaller queries or no queries are executed. This is problematic because the unused capacity could be used to cache more table data, thereby improving throughput for the transactional portion of the workload. Furthermore, configuring the buffer pool to an adequate size can be time consuming and error prone, and can cause queries to fail or have to spill to disk if too little memory capacity is left for temporary data.
[0005] The present disclosure relates to techniques for improving memory allocation in a database management system. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 A system for collaborative memory management according to an embodiment is shown.
[0007] Figure 2 A method according to an embodiment is shown.
[0008] Figure 3 Another example computer system including collaborative memory management according to an embodiment is shown.
[0009] Figure 4A An example query execution plan is shown according to an embodiment.
[0010] Figure 4B The steps performed during query execution according to an embodiment are shown.
[0011] Figure 4C An example of reallocating table data and temporary data in a storage buffer according to an embodiment is shown.
[0012] Figure 5 The hardware of a special purpose computing machine configured according to the above disclosure is shown. DETAILED DESCRIPTION
[0013] Techniques for collaborative memory management are described herein. In the following description, for the purpose of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of some embodiments. Various embodiments defined by the claims may include some or all of the features in these examples, alone or in combination with other features described below, and may also include modifications and equivalents of the features and concepts described herein.
[0014] Figure 1 A system for collaborative memory management according to an embodiment is shown.For example, features and advantages of the present disclosure include techniques for dynamically adjusting memory allocation between table data and temporary data for queries in a database management system. Figure 1 A computer system 100 is shown running one or more software applications 101a-n and a database management system 102 on one or more processors 103. Applications 101-an can store and retrieve application data to one or more databases 151 stored in a storage medium 104 (e.g., solid-state memory or a disk drive). Data stored in a database is typically stored in tables. A database management system (DBMS) is a software system for creating and managing databases. A DBMS enables end users to create, protect, read, update, and delete data in a database. The DBMS 102 serves as an interface between the database 151 and the user or application 101a-n, ensuring that the data is consistently organized and remains easily accessible. Thus, the database management system 102 receives queries from the applications 101a-n and executes the queries to retrieve data stored in the database 151.
[0015] Computer system 100 also includes memory 105, such as random access memory (e.g., RAM, DRAM, or inline memory modules such as DIMMs). Memory 105 is sometimes referred to as main memory. Because transactions with storage medium 104 are slower than transactions with memory 105, DBMS 102 can be provided with allocations in memory 105 for storing data (usually table data) from database 151 and temporary data required for executing queries. The memory allocated to DBMS is sometimes referred to as, for example, storage buffers. As described above, the memory provided to DBMS is usually allocated between a memory for storing table data and a memory for storing temporary data. Typically, the memory allocated for table data and the memory allocated for temporary data are predefined and static. However, using the technology described herein, the portion of the storage buffer allocated to table data and temporary data can be changed on a query-by-query basis, or even changed during the execution of different parts of a single query. This leads to more efficient utilization of storage buffers and improvements in performance and cost efficiency of DBMS.
[0016] Embodiments of the present disclosure include a DBMS 102 that includes a memory manager 112 for controlling the interaction between the DBMS 102 and a storage buffer 152 in memory 105. The memory manager 112 is a software component that dynamically handles allocation / deallocation of both table data and temporary data (e.g., on a query-by-query basis or even multiple times during a single query), where the memory for these allocations / deallocations of both table data and temporary data is from a common storage buffer 152 available to the memory manager 112 for its data (e.g., allocated to the memory manager 112 by the DBMS 102). The memory manager 112 may, for example, specify table data allocations 120 and temporary data allocations 121 in the storage buffer 152, for example, where the combination of the table data allocations and temporary data allocations does not exceed the total memory allocation limit of the memory available to the storage buffer 152 in the memory 105. The DBMS 102 may retrieve the table data from the database 151 and instruct the memory manager 112 to store the table data in the table data allocations 120 of the storage buffer 152. Additionally, when executing queries, for example, DMBS 102 may instruct memory manager 112 to store and retrieve temporary data in temporary data allocation 121 of storage buffer 152. Advantageously, memory manager 112 may change table data allocations and temporary data allocations in response to changing requirements of one or more queries. Thus, allocations 120 and 121 may be modified “on the fly” by memory manager 112, as shown at 150, for example, where the amount of memory available for temporary data may increase or decrease over time based on query execution requirements.
[0017] For example, DBMS 102 may have an initial table data memory allocation 120 and a temporary data memory allocation 121. When DBMS 102 receives a new query, DBMS 102 may reduce allocation 120 and increase allocation 121 to provide more storage buffer space for query execution. DBMS 102 may determine the amount of memory used to process one or more portions of the query. When new temporary data from a query causes the combination of table data and temporary data to exceed the total memory allocation limit of storage buffer 152, DBMS 102 may instruct memory manager 112 to change allocations 120 and 121 to accommodate the additional temporary data required to process the various portions of the query. In response to the instructions, memory manager 112 may modify the table data memory allocation and the temporary data memory allocation. Memory manager 112 may reduce table data memory allocation 120 (e.g., and delete table data as needed) and increase temporary data memory allocation 121 (e.g., to store temporary data for the query, such as a hash table). Storing data to storage buffer 152 and deleting data from storage buffer 152 are sometimes referred to as "caching" and "evicting" data. Thus, when the combination of the table data and the amount of memory used to process one or more portions of a query is greater than the total memory allocation limit, the allocation is dynamically changed so that the combination of the table data in the storage buffer 152 and the amount of the storage buffer 152 required to store temporary data used to process the query (or portions thereof) is less than the total memory allocation limit for the storage buffer 152. Once the new table data and temporary data allocations are made, the memory manager 152 may store the temporary data (or portions thereof) generated by the query in the storage buffer 152.
[0018] Figure 2A method according to an embodiment is shown. At 201, a DBMS may store table data from a database in a storage buffer of a random access memory. The storage buffer may include a table data storage allocation for storing table data, a temporary data storage allocation for storing temporary data associated with one or more queries, and a total memory allocation limit. At 202, a DMBS may receive a query. The DBMS may generate an execution plan for the query, which may involve, for example, a plurality of different temporary memory requirements at different execution points of the query. At 203, the DBMS may determine an amount of memory for processing at least a portion of the query (e.g., an initial portion, a subsequent portion, and / or the total query). At 204, the DBMS instructs a memory manager to modify the table data storage allocation and the temporary data storage allocation. For example, when the combination of the table data and the amount of memory for processing at least a portion of the query is greater than the total memory allocation limit, the memory manager may reduce the table data storage allocation and increase the temporary data storage allocation so that the combination of the table data and the temporary data in the storage buffer is less than the total memory allocation limit of the storage buffer. Reducing the table data storage allocation may include deleting the table data from the storage buffer at 205. At 206, temporary data generated by the query (or a particular portion of the query) is stored in a storage buffer. At 207, when the query (or portion of the query) completes processing, the temporary data can be deleted from the storage buffer, for example. At 208, the system determines whether there are more portions of the query to be processed. If so, and the query is complete, the process returns to 203 and repeats steps 204-207. If not, the memory used for temporary data can generally be available again for caching table data or temporary data for subsequent queries. Therefore, at 209, the temporary data memory allocation in the storage buffer is modified for the next query. Subsequent queries may require more or less temporary data. If subsequent queries use less temporary data, the table data memory allocation in the storage buffer can be increased, for example.
[0019] Figure 3 Another example computer system including collaborative memory management according to an embodiment is shown. In this example, DBMS 102 also includes a query execution engine 310 and a storage manager 311, and the memory manager includes a memory allocator 313 for allocating memory through the operating system 306. DBMS 102 can interface with the storage manager 311 to store table data to and retrieve table data from the database (or multiple) 151. The memory manager 112 can interface with the storage manager 311 to store table data in the database 151 to Figure 1 The storage buffer 105 in Figure 1The memory allocator 313 may communicate with the operating system 306 to allocate memory in the memory buffers. For example, the memory allocator 313 may communicate with the operating system 306 to set a total memory allocation limit for the memory buffers, table data memory allocations, and temporary data memory allocations.
[0020] The DBMS 102 may instruct the storage manager 311 to retrieve one or more data tables and send them to the memory manager 112. The DBMS 102 then instructs the memory manager 112 to store the table data in a portion of the storage buffer allocated for the table data. When the DBMS 102 receives a query, temporary data associated with the query may be stored in the portion of the storage buffer allocated for the temporary data. An example table data memory allocation 120a and temporary data memory allocation 121a are shown for the storage buffer 305a in an initial state. When another query is received, the DBMS 102 may determine that the temporary data memory allocation 121a is insufficient to process at least a portion of the query. For example, when a query is received, the query execution engine 310 may analyze the query and generate a query execution plan. The query execution plan may require different amounts of temporary data at different points in time during query execution. The DBMS 102 may determine that the initial portion of the query requires more temporary memory than is currently allocated. Therefore, the memory allocator 313 may increase the amount of temporary data memory allocation from 121a to 121b, and reduce the amount of table data memory allocation from 120a to 120b. Prior to reducing the table data memory allocation 120a, the memory manager 112 may delete the table data from the storage buffer. Storage buffer 305b shows the state of the storage buffer after the new allocation is applied. The change in the storage buffer allocation is shown at 350. Once the initial portion of the query is completed, the DBMS 102 may instruct the memory manager 112 to delete the temporary data associated with the initial portion of the query. One or more subsequent portions of the query (if any) may be analyzed to determine the amount of temporary data to process each subsequent portion of the query, and the temporary data memory allocation may be modified to accurately accommodate each amount of temporary data required for the current processing operation. Therefore, during the execution of the query plan, the table data memory allocation and the temporary data memory allocation may be advantageously modified dynamically to optimize the use of available storage buffer space for both table data and temporary data.
[0021] Thus, storage buffers are controlled to implement collaborative memory management that exploits the low-overhead nature of the vmcache approach, for example, extending the memory managed by the buffer pool from only cached table data to also include intermediate results, thereby breaking down the barriers between these memory consumers that exist in traditional systems. These techniques can manage all main memory in a way that leaves no unused memory, and enable on-demand allocation of temporary data that approaches or reaches memory limits. For example, the present disclosure can solve the problems of inefficient memory usage and complex storage buffer configuration, and can avoid making the memory usage of different DBMS components configurable as a whole. Instead, the system can indiscriminately serve memory requests for cached table data or temporary data until the memory limit is reached.
[0022] For example, the memory limit (or total memory allocation limit) can be the available DRAM capacity or a user-configured limit. When the limit is reached and a new request arrives, the system should evict cached table data in order to free up enough memory to satisfy the request. In this way, in the case of sufficient memory demand, memory is always fully utilized to cache table data when temporary data does not require memory, but memory for temporary data can still be allocated at any point in time up to the memory limit. This offloads the burden of configuring memory areas (e.g., buffer pool size) from the system user and maximizes the utility obtained from the available memory capacity by always using memory that is not currently used for other purposes to cache table data.
[0023] Embodiments of collaborative memory management may be enabled by a central memory manager (described above) that tracks the amount of memory allocated to cache table data and temporary data, and handles requests for either type of memory. Conceptually, a memory manager includes memory buffer management and the ability to also service memory requests for temporary data. The memory manager may handle all allocations and deallocations in the system and ensure that the total memory allocated to both cache table data and temporary data never exceeds the memory limit. In various embodiments, the memory manager provides the following interface to the rest of the system:
[0024] pin(pid): Binds a table data page and loads it from storage if it is not already in memory.
[0025] up(pid): Unbind the table data page.
[0026] AllocateTemporal(size): Allocates memory for temporary data.
[0027] freeTemporary(pointer,size): Frees memory previously used for temporary data, which can be reused for cache table data.
[0028] Access to data pages can be synchronized between threads, and the memory manager provides at least two variants of pin() and unpin() calls, one for shared read-only locks and one for exclusive write locks. When servicing a request for memory (i.e., a pin() or allocateTemporary() call), the memory manager can simply allocate new memory using the system allocator as long as the total memory usage does not exceed the limit. Once the limit is reached, it must evict (e.g., using an eviction policy) table data pages and free their memory until there is enough space for new allocations.
[0029] In one example embodiment, the only case where an allocateTemporary() call cannot be processed occurs when the combination of a new request, already allocated temporary memory, and bound data pages exceeds the memory limit. However, this occurs much later in the case of the collaborative approach than in a conventional system, where, for example, data pages may not be evicted to dynamically make room for temporary data.
[0030] For example, the memory manager functionality may include specialized internal stapling of pages on allocateTemporary() calls, and unstapling of those pages for freeTemporary() calls. This allows buffer frames used for temporary data to be reused for table data after use. Thus, some embodiments of the system may perform all large temporary allocations through the memory manager. With such an implementation, for example, the allocation granularity of temporary data may be a multiple of the memory manager's page size.
[0031] Figure 4A An example query execution plan according to an embodiment is shown. In this example, the DBMS may receive the following query:
[0032] SELECT COUNT(*)FROM R,S WHERE R.key=S.key.
[0033] The execution plan for query 401 involves a JOIN operator where R.key=S.key 402 , which requires retrieving pages of table R 403 and table S 404 . Figure 4BSteps performed during query execution according to an embodiment are shown. At 410, a query is received and an execution plan is generated. Temporary memory for executing the query may be provided to a memory manager. At 411, the memory manager allocates a portion of the storage buffer sufficient to process the query. For example, table data may be deleted to accommodate more temporary data. At 412, the DBMS reads table R and creates a hash table for joining tuples from R with tuples from S. At 413, the memory manager stores the hash table in the portion of the storage buffer allocated for temporary data. At 414, table R is scanned and pages are bound / unbound (as needed). At 415, table S is scanned and pages are bound / unbound (as needed) to perform the join. The system may probe R's hash table to obtain query results (e.g., by counting tuples with equal keys). At 416, the query is completed, temporary data for the query is deleted, and temporary data memory allocations for the query may be released or otherwise removed, for example.
[0034] Figure 4C An example of reallocating table data and temporary data in a storage buffer according to an embodiment is shown. Various allocations of storage buffers during different parts of query execution are shown here. Initially, at 420, the storage buffer includes table data but not temporary data. At 421, it is determined that more memory is needed to store the temporary data for the query. For example, the DBMS may determine that 30% of the storage buffer is required for temporary data. The DBMS may issue an allocateTemporary command to the memory manager to allocate storage buffers. Therefore, the temporary data memory allocation is increased, and the table data allocation is decreased. In this example, the table data is deleted from the storage buffer. At 422, the temporary data is stored in the storage buffer. Reference Figure 4A and Figure 4B In a query in, for example, table R can be scanned by sequentially binding / unbinding pages of table R. At 423, pages of tables R and S are pinned / unbinded in the table data portion of the storage buffer as needed, and temporary data is stored and deleted in the temporary data portion of the storage buffer as needed. The pin command can check whether the page is already cached in the storage buffer. If not, the pin command can cause the memory manager to evict another (unpinned) cached page of the table data and load the page from disk. For example, unpinning marks a cached page of table data, which can be deleted after its data has been read and processed. At 424, the query is completed and the temporary data is deleted. At 425, the table data memory allocation is increased, and the temporary data memory allocation is decreased. After the query is completed, the allocation can depend on the next query or other activities of the DBMS.
[0035] Figure 5The hardware of a special-purpose computing machine configured according to the above disclosure is shown. The following hardware description is only an example. It should be understood that various computer topologies can be used to implement the above technology. Figure 5 An example computer system 510 is shown in FIG. The computer system 510 includes a bus 505 or other communication mechanism for transmitting information, and one or more processors 501 coupled to the bus 505 for processing information. The computer system 510 also includes a memory 502 coupled to the bus 505 for storing information and instructions to be executed by the processor 501, including, for example, information and instructions for performing some of the above-mentioned techniques. The memory 502 can also be used to store programs executed by the processor 501. Possible implementations of the memory 502 can be, but are not limited to, random access memory (RAM), read-only memory (ROM), or both. A storage device 503 for storing information and instructions is also provided. Common forms of storage devices include, for example, hard drives, magnetic disks, optical disks, CD-ROMs, DVDs, solid-state disks, flash memory or other non-volatile memories, USB memory cards, or any other electronic storage media from which a computer can read. For example, the storage device 503 may include source code, binary code, or software files for performing the above-mentioned techniques. The storage device 503 and the memory 502 are both examples of non-temporary computer-readable storage media (also referred to as storage media).
[0036] In some systems, computer system 510 may be coupled to a display 512 via bus 505 for displaying information to a computer user. An input device 511, such as a keyboard, touch screen, and / or mouse, is coupled to bus 505 for communicating information and command selections from the user to processor 501. The combination of these components allows the user to communicate with the system. In some systems, bus 505 represents, for example, multiple dedicated buses used to couple together various components of a computer.
[0037] The computer system 510 also includes a network interface 504 coupled to the bus 505. The network interface 504 can provide two-way data communication between the computer system 510 and the local network 520. The network 520 can represent one or more networking technologies, such as Ethernet, a local wireless network (e.g., WiFi), or a cellular network. For example, the network interface 504 can be a wireless or wired connection. For example, the computer system 510 can send and receive information to the Internet 530 across a wired or wireless local area network, an intranet, or a cellular network through the network interface 504. In some embodiments, the front end (e.g., a browser) can, for example, access data and features on a back-end software system, which can reside on multiple different hardware servers 531 or reside on servers 532-535 across the Internet 530. For example, one or more of the servers 532-535 can also reside in a cloud computing environment.
[0038] Other embodiments
[0039] Each of the following non-limiting features in the following embodiments may exist independently, or may be combined with one or more other features in the following embodiments in various arrangements or combinations. In various embodiments, the present disclosure may be implemented as a system, method, or computer-readable medium.
[0040] Various embodiments of the present disclosure include methods, computer systems and / or non-transitory computer-readable media, wherein the computer system includes at least one processor, a random access memory and at least one non-transitory computer-readable medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the computer system to perform the method, and the non-transitory computer-readable medium stores computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, perform the method.
[0041] In one embodiment, the present disclosure includes a method for managing memory in a database management system, the method comprising: storing, by a database management system executed on a computer system, table data from a database in a random access memory, the random access memory having a database management system storage buffer, the database management system storage buffer including a table data storage allocation for storing the table data, a temporary data storage allocation for storing temporary data associated with one or more queries, and a total memory allocation limit; receiving a query in the database management system; determining, in the database management system, an amount of memory for processing at least a portion of the query; modifying, by a memory manager, the table data storage allocation and the temporary data storage allocation, wherein when a combination of the table data and the amount of memory for processing at least a portion of the query is greater than the total memory allocation limit, the memory manager reduces the table data storage allocation and increases the temporary data storage allocation so that a combination of the table data in the storage buffer and the amount of the storage buffer for processing at least a portion of the query is less than the total memory allocation limit; and storing, by the memory manager, the temporary data generated by at least a portion of the query in the storage buffer.
[0042] In one embodiment, in response to instructions from a database management system, the memory manager: stores a total memory allocation for the database management system; reduces or increases a table data memory allocation; stores table data in random access memory and deletes table data from the random access memory; and stores temporary data in random access memory and deletes temporary data from the random access memory.
[0043] In one embodiment, the method further comprises upon completion of the query: deleting, by the memory manager, temporary data associated with the at least a portion of the query; and increasing, by the memory manager, the table data memory allocation.
[0044] In one embodiment, the portion of the query is an initial portion of the query, and the method further includes: determining a second amount of memory for processing one or more subsequent portions of the query; and for one or more subsequent portions of the query: reducing the table data memory allocation so that the combination of the table data and the amount of memory for processing the specific subsequent portion of the query is less than the total memory allocation limit; or increasing the table data memory allocation so that the combination of the table data and the amount of memory for processing the specific subsequent portion of the query is less than the total memory allocation limit.
[0045] In one embodiment, the portion of the query includes a JOIN function for logically connecting at least two data tables stored at least partially in random access memory, and wherein one or more subsequent portions of the query include one or more JOIN functions for connecting one or more additional data tables stored at least partially in random access memory.
[0046] In one embodiment, reducing the table data memory allocation includes deleting, by the memory manager, a portion of the table data from the memory buffer.
[0047] In one embodiment, a database management system includes a memory allocator for communicating with an operating system executing on a computer system, wherein a memory manager communicates with the operating system through the memory allocator to decrease or increase table data memory allocation.
[0048] In one embodiment, the temporary data is a data structure storing key-value pairs.
[0049] In one embodiment, the data structure is a hash table.
[0050] In one embodiment, the query is a SQL query.
[0051] The above description shows various embodiments and examples of how aspects of some embodiments may be implemented. The above examples and embodiments should not be considered to be the only embodiments, and are presented to illustrate the flexibility and advantages of some embodiments defined by the appended claims. Based on the above disclosure and the appended claims, other arrangements, embodiments, implementations and equivalents may be adopted without departing from the scope of the invention defined by the claims.
Claims
1. A method for managing memory in a database management system, include: storing, by a database management system executing on a computer system, table data from a database in a random access memory having a database management system storage buffer, the database management system storage buffer including a table data memory allocation for storing the table data, a temporary data memory allocation for storing temporary data associated with one or more queries, and a total memory allocation limit; receiving a query in a database management system; determining, in a database management system, an amount of memory used to process at least a portion of the query; modifying, by the memory manager, a table data memory allocation and a temporary data memory allocation, wherein, when a combination of the table data and an amount of memory used to process at least a portion of a query is greater than a total memory allocation limit, the memory manager decreases the table data memory allocation and increases the temporary data memory allocation such that a combination of the table data in the memory buffer and an amount of the memory buffer used to process at least a portion of the query is less than the total memory allocation limit; as well as Temporary data generated by at least the portion of the query is stored, by the memory manager, in a memory buffer.
2. The method according to claim 1, in, The storage manager responds to instructions from the database management system: Total memory allocation for storing database management systems; Reduce or increase table data storage allocation; storing table data into random access memory and deleting table data from random access memory; as well as The temporary data is stored in the random access memory and the temporary data is deleted from the random access memory.
3. The method according to claim 1, further comprising: include: When the query is completed: deleting, by the memory manager, temporary data associated with the at least a portion of the query; as well as Increase table data memory allocation by the memory manager.
4. The method according to claim 1, in, The portion of the query is an initial portion of the query, the method further comprising: determining a second amount of memory for processing one or more subsequent portions of the query; and For the one or more subsequent parts of the query: reducing the table data memory allocation so that the combination of the table data and the amount of memory used to process a particular subsequent portion of the query is less than the total memory allocation limit; or Increase the table data memory allocation so that the combination of the table data and the amount of memory used to process a particular subsequent portion of the query is less than the total memory allocation limit.
5. The method according to claim 4, in, The portion of the query includes a JOIN function for logically connecting at least two data tables stored at least partially in random access memory, and wherein the one or more subsequent portions of the query include one or more JOIN functions for connecting one or more additional data tables stored at least partially in random access memory.
6. The method according to claim 1, in, Reducing the table data memory allocation includes deleting, by the memory manager, a portion of the table data from the memory buffer.
7. The method according to claim 1, in, The database management system includes a memory allocator for communicating with an operating system running on the computer system, wherein the memory manager communicates with the operating system through the memory allocator to decrease or increase the table data memory allocation.
8. The method according to claim 1, in, Temporary data is a data structure that stores key-value pairs.
9. The method according to claim 8, in, The data structure is a hash table.
10. The method according to claim 1, in, Query is a SQL query.
11. A computer system, include: at least one processor; Random Access Memory; At least one non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one processor, cause a computer system to perform a method comprising: storing, by a database management system executing on a computer system, table data from a database in a random access memory having a database management system storage buffer, the database management system storage buffer including a table data memory allocation for storing the table data, a temporary data memory allocation for storing temporary data associated with one or more queries, and a total memory allocation limit; receiving a query in a database management system; determining, in a database management system, an amount of memory used to process at least a portion of the query; modifying, by the memory manager, the table data memory allocation and the temporary data memory allocation, wherein when a combination of the table data and an amount of memory used to process at least a portion of a query is greater than a total memory allocation limit, the memory manager decreases the table data memory allocation and increases the temporary data memory allocation such that a combination of the table data in the memory buffer and an amount of the memory buffer used to process at least a portion of the query is less than the total memory allocation limit; and Temporary data generated by at least the portion of the query is stored, by the memory manager, in a memory buffer.
12. The computer system according to claim 11, in, The storage manager responds to instructions from the database management system: Total memory allocation for storing database management systems; Reduce or increase table data storage allocation; storing table data into random access memory and deleting table data from random access memory; as well as The temporary data is stored in the random access memory and the temporary data is deleted from the random access memory.
13. The computer system of claim 11, further comprising, upon completing the query: deleting, by the memory manager, temporary data associated with the at least a portion of the query; and Increase table data memory allocation by the memory manager.
14. The computer system according to claim 11, in, The portion of the query is an initial portion of the query, the method further comprising: determining a second amount of memory for processing one or more subsequent portions of the query; and For the one or more subsequent parts of the query: reducing the table data memory allocation so that the combination of the table data and the amount of memory used to process a particular subsequent portion of the query is less than the total memory allocation limit; or Increase the table data memory allocation so that the combination of the table data and the amount of memory used to process a particular subsequent portion of the query is less than the total memory allocation limit.
15. The computer system according to claim 11, in, Reducing the table data memory allocation includes deleting, by the memory manager, a portion of the table data from the memory buffer.
16. A non-transitory computer readable medium storing computer executable instructions, which when executed by at least one processor perform a method, the method include: storing, by a database management system executing on a computer system, table data from a database in a random access memory having a database management system storage buffer, the database management system storage buffer including a table data memory allocation for storing the table data, a temporary data memory allocation for storing temporary data associated with one or more queries, and a total memory allocation limit; receiving a query in a database management system; determining, in a database management system, an amount of memory used to process at least a portion of the query; modifying, by the memory manager, a table data memory allocation and a temporary data memory allocation, wherein, when a combination of the table data and an amount of memory used to process at least a portion of a query is greater than a total memory allocation limit, the memory manager decreases the table data memory allocation and increases the temporary data memory allocation such that a combination of the table data in the memory buffer and an amount of the memory buffer used to process at least a portion of the query is less than the total memory allocation limit; as well as Temporary data generated by at least the portion of the query is stored, by the memory manager, in a memory buffer.
17. The non-transitory computer readable medium of claim 16, in, The storage manager responds to instructions from the database management system: Total memory allocation for storing database management systems; Reduce or increase table data storage allocation; storing table data into random access memory and deleting table data from random access memory; as well as The temporary data is stored in the random access memory and the temporary data is deleted from the random access memory.
18. The non-transitory computer-readable medium of claim 16, further comprising, upon completing the query: deleting, by the memory manager, temporary data associated with the at least a portion of the query; and Increase table data memory allocation by the memory manager.
19. The non-transitory computer readable medium of claim 16, further comprising: include: determining a second amount of memory for processing one or more subsequent portions of the query; as well as For the one or more subsequent parts of the query: reducing table data memory allocation so that a combination of the table data and an amount of memory used to process a particular subsequent portion of the query is less than a total memory allocation limit; or Increase the table data memory allocation so that the combination of the table data and the amount of memory used to process a particular subsequent portion of the query is less than the total memory allocation limit.
20. The non-transitory computer readable medium of claim 16, in, Reducing the table data memory allocation includes deleting, by the memory manager, a portion of the table data from the memory buffer.
