Memory management method, system, electronic device, storage medium and program product

Abstract

The present application discloses a memory management method, system, electronic device, storage medium and program product, which relate to the field of computer technology. The method includes: presetting keyword information in a memory module, the keyword information is used to identify the memory level type and usage category to which the memory module belongs; establishing a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator in the memory allocator, and based on the mapping relationship, retrieving and recording physical status information; and using the memory allocator to classify, manage and recycle memory modules with different keyword information according to the physical status information. The present application binds the memory management unit of the memory allocator to the memory module through the established mapping relationship, classifies, manages and recycles memory of different levels and types, and alleviates the adaptive memory management of the upper-level database system under different service pressures.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a memory management method, system, electronic device, storage medium, and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that what is described herein is prior art by its inclusion in this section.

[0003] Cloud-native databases are increasingly popular among internet and broader internet users. The rapid improvement in user services is also putting pressure on database memory management. Currently, cloud-native databases are primarily developed based on older open-source databases. This presents a challenge: Memory management in these early open-source databases assumes a distributed system deployment model, such as sharding, which replaces commercial databases with a surge in services. Furthermore, they assume infinite memory, resulting in a crude approach to memory management. While simple unified block allocation and buffer pool cache management mechanisms exist, they lack effective unified memory management for complex SQL and concurrent SQL queries. The underlying memory layer relies on the operating system or the database's built-in memory allocator embedded in the bundle. Even if memory is released on the database side, the allocator is unaware of the upper-layer database semantics, and actual physical memory is not reclaimed.

[0004] The current approach in the general cloud native industry is to find a well-suited and powerful managed memory allocator to cover most scenarios. Since cloud native databases currently use a standard memory configuration, exceeding the configured memory usage will immediately trigger a hard landing via the out-of-memory killer (oomkiller), rendering high availability (HA) unavailable and leading to customer service issues. Standard database configurations include memory levels at the instance, library, session, and structured query language (SQL) levels. For example, the MySQL database currently encapsulates allocation functions for the memory manager. However, except for the allocation of the very large memory buffer pool, which is directly allocated by the operating system, all other memory allocations are made through a third-party memory allocator. Because the third-party memory allocator and the database cannot perceive each other's actual information, physical memory cannot be released immediately when services surge and memory allocation and release are frequent, and different memory levels cannot be classified, managed, and recycled. Summary of the Invention

[0005] The embodiments of the present application provide a memory management method, system, electronic device, storage medium and program product to at least solve the problem in the prior art that different memory levels and types of memories cannot be classified, managed and recycled because the third-party memory allocator and the database cannot perceive each other's actual information.

[0006] According to one aspect of the present application, a memory management method is also provided, comprising:

[0007] Presetting keyword information in the memory module, the keyword information is used to identify the memory level type and usage category to which the memory module belongs;

[0008] Establishing a mapping relationship between the keyword information and a memory management unit corresponding to the memory allocator in a memory allocator, so as to retrieve and record the physical state information of the memory management unit based on the mapping relationship when the memory module requests memory through the memory management unit;

[0009] According to the physical status information, the memory allocator is used to classify, manage and recycle the memory modules with different keyword information.

[0010] In some embodiments, the keyword information includes a first keyword and a second keyword; the first keyword is used to identify a memory class type to which the memory module belongs, and the second keyword is used to identify a usage category of the memory module, wherein the step of establishing a mapping relationship between the keyword information and a memory management unit corresponding to the memory allocator includes:

[0011] A first mapping relationship is established between the first keyword and the second keyword, and a second mapping relationship is established between the second keyword and the memory management unit. Then, in the process of the memory module requesting memory through the memory management unit, the physical status information of the memory management unit is retrieved and recorded based on the first mapping relationship and the second mapping relationship.

[0012] In some embodiments, the first keyword is placed in an index block header of the memory module. Then, during a process in which the memory module requests memory through the memory management unit, the step of retrieving and recording the physical state information of the memory management unit based on the first mapping relationship and the second mapping relationship includes:

[0013] During the process of the memory module requesting memory through the memory management unit, the first keyword placed in the index block header is intercepted, and the physical status information is retrieved and recorded based on the first keyword and the first mapping relationship and the second mapping relationship.

[0014] In some embodiments, the step of using the memory allocator to classify, manage, and recycle the memory modules with different keyword information includes:

[0015] For the memory modules with different keyword information, the memory allocator is used to adopt different memory management strategies for classification management and recycling.

[0016] In some embodiments, the memory level type to which the memory module belongs includes instance level and / or session level. After retrieving and recording the physical state information of the memory management unit, the method further includes:

[0017] determining the memory usage of the memory modules having different keyword information according to the physical status information;

[0018] Then, the step of using the memory allocator to classify, manage, and recycle the memory modules with different keyword information according to the physical status information includes:

[0019] The memory allocator adopts a watermark management strategy, and the memory management unit classifies and manages and recycles the corresponding memory modules based on the keyword information. The watermark management strategy is a strategy for classifying and managing and reclaiming the memory modules when the memory usage exceeds a preset usage warning value.

[0020] In some embodiments, the level type to which the memory module belongs includes one or more of an instance level, a session level, and a Structured Query Language (SQL) level, and the step of classifying, managing, and recycling the memory modules with different keyword information using the memory allocator according to the physical status information further includes:

[0021] The memory allocator is used to adopt a periodic cleaning strategy, and the memory management unit periodically cleans the corresponding memory module based on the keyword information to release physical memory. The periodic cleaning strategy is a strategy for cleaning the memory module according to a preset recycling cycle.

[0022] According to another aspect of the present application, a memory management system is provided, including:

[0023] A keyword adding module, used for presetting keyword information in the memory module, wherein the keyword information is used for identifying the memory level type and usage category to which the memory module belongs;

[0024] a mapping module, configured to establish, in a memory allocator, a mapping relationship between the keyword information and a memory management unit corresponding to the memory allocator, so as to retrieve and record the physical state information of the memory management unit based on the mapping relationship when the memory module requests memory through the memory management unit;

[0025] A recycling management module is used to classify, manage and recycle the memory modules with different keyword information using the memory allocator according to the physical status information.

[0026] In some embodiments, the keyword information includes a first keyword and a second keyword; the first keyword is used to identify a memory class type to which the memory module belongs, and the second keyword is used to identify a usage category of the memory module, wherein the step of the mapping module establishing a mapping relationship between the keyword information and a memory management unit corresponding to the memory allocator includes:

[0027] A first mapping relationship is established between the first keyword and the second keyword, and a second mapping relationship is established between the second keyword and the memory management unit. Then, in the process of the memory module requesting memory through the memory management unit, the physical status information of the memory management unit is retrieved and recorded based on the first mapping relationship and the second mapping relationship.

[0028] In some embodiments, the first keyword is placed in an index block header of the memory module, and the system further comprises:

[0029] A data interception module is used to intercept the first keyword placed in the index block header during the process of the memory module requesting memory through the memory management unit, and to retrieve and record the physical status information based on the first keyword based on the first mapping relationship and the second mapping relationship.

[0030] In some embodiments, the memory allocator includes a Jemalloc memory allocator, and the memory management unit is a concurrent allocation unit arena of the Jemalloc memory allocator.

[0031] According to another aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method steps when executing the computer program.

[0032] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method steps when executed by a processor.

[0033] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method steps are implemented.

[0034] In an embodiment of the present application, the memory management unit inside the memory allocator is bound to the memory module through an established mapping relationship, so that the memory allocator and the upper-level system with the corresponding memory module can obtain each other's actual information based on the mapping relationship, thereby classifying and managing and recycling memories of different memory levels and types, alleviating the adaptive memory management of upper-level systems such as database systems under different service pressures, and ensuring the timely release of real physical memory when the database service pressure increases. For database scenarios with decentralized kernel memory management, only the memory modules that need to be adjusted according to the service are bound to the memory management unit according to actual memory management needs, ensuring the availability of the instance while improving the efficiency of memory management and recycling. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0036] Figure 1 It is a schematic diagram of the overall memory allocation and management of the database;

[0037] Figure 2 This is a diagram showing the current status of memory usage in the relational database management system MySQL;

[0038] Figure 3 is a flowchart of a memory management method according to an embodiment of the present application;

[0039] Figure 4 Schematic diagram of the mapping relationship between the first keyword key and the second keyword classification key according to an embodiment of the present application;

[0040] Figure 5 2 is a schematic diagram of a mapping relationship between a memory module having a second keyword classification key and a concurrent allocation unit according to an embodiment of the present application;

[0041] Figure 6 2 is a schematic diagram of calling physical status information of a concurrent allocation unit (arena) according to an embodiment of the present application;

[0042] Figure 7This is a diagram of the overall configuration strategy architecture of the cloud-native relational database PolarDB MySQL using the memory management method of the embodiment of the present application;

[0043] Figure 8 is a schematic diagram of a water level management strategy according to an embodiment of the present application;

[0044] Figure 9 2 is a schematic diagram of memory module memory management at the session level according to an embodiment of the present application;

[0045] Figure 10 Schematic diagram of a memory management system according to an embodiment of the present application.

[0046] The above drawings include the following reference numerals:

[0047] 10. Keyword addition module; 20. Mapping module; 30. Memory allocator; 40. Recycling management module; 50. Data interception module. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] The open-source relational database management system MySQL and cloud-native and distributed database products based on MySQL / PostgreSQL, such as GaussDB (for MySQL), Aurora, and TDSQL-C for MySQL, all rely on operating system or third-party memory allocators. However, these memory allocators lack awareness of the database's semantic memory lifecycle. This results in instances where physical memory cannot be fully released even after the database layer releases memory, leading to hard kills by the out-of-memory killer (oomkiller), resulting in instance unavailability. Currently, commercial databases implement their own internal management directly at the data layer, independent of the operating system. This solution is unsuitable for the decentralized upper-layer kernel memory management found in cloud-native databases, and the cost of retrofitting is extremely high.

[0051] Typically, the overall memory allocation and management of the database is as follows Figure 1 As shown, the instance-level shared memory includes:

[0052] Data Page, Lock Info, Adaptive Hash, Index Page, Dict Info, Insert Buffer, Key Buffer, Query Cache, Thread Cache, Table Cache, Redo Log Buffer, Binlog Buffer, Table_definition_cache, and Additional Memory Pool

[0053] Innodb_additional_mem_pool etc.

[0054] Instance-level management of shared memory includes:

[0055] Performance data Performance Schemas, monitoring data Monitors, and execution plan management SPM.

[0056] Session-level private memory includes:

[0057] Thread stack, sort buffer, join buffer, read buffer, random read buffer, network buffer size, temporary table, and bulk insert buffer, etc.

[0058] Based on the aforementioned memory allocation, if the upper-level database cannot fully understand the actual situation of the lower-level memory allocator, the actual physical memory that should be released may not be released, thus affecting database service performance. For example, the MySQL database currently encapsulates the allocation function of the memory manager. Except for the allocation of the ultra-large memory buffer pool, which is directly allocated by the operating system, all other memory allocations are made through a third-party memory allocator.

[0059] like Figure 2This article describes the current memory usage status of MySQL, a relational database management system. MySQL is a pluggable database, divided into a server query layer and an InnoDB storage engine layer. For example, the eight modules within the thread-level cache of the server query layer are all managed as session-level caches. The data buffer pool, additional memory pool, and log memory cache of the InnoDB storage engine layer, as well as the query result cache, index cache, table object cache, and table metadata cache of the server query layer, are all managed according to a unified global cache management method.

[0060] However, because the memory allocator isn't aware of database issues, physical memory often can't be released immediately when services surge and memory allocation and release become frequent. Since cloud-native databases currently use standard memory configurations, exceeding the configured memory usage triggers a hard landing via the OOMKiller mechanism, causing high availability to fail.

[0061] In view of the above problems, please see Figure 3 A first embodiment of the present invention provides a memory management method, the method comprising the following steps:

[0062] Step S11: Preset keyword information in the memory module. The keyword information is used to identify the memory level type and usage category to which the memory module belongs. For example, the keyword information is used to identify whether the corresponding memory module is at the instance level or the session level, and whether the memory module is used for sorting or created for a temporary table. In view of the actual situation of decentralized upper-level kernel memory management in cloud-native databases, keyword information can be set only for some memory modules. Later, based on the keyword information, the corresponding memory modules are classified, managed, and recycled according to level types such as instance, library, session, and SQL statement.

[0063] Step S12: Establish a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator in the memory allocator, so that in the process of the memory module requesting memory through the memory management unit, based on the mapping relationship, the physical status information of the memory management unit is retrieved and recorded. The physical status information of the memory management unit includes the actual amount of physical memory allocated to each memory module and the currently adopted memory management strategy, etc.

[0064] Step S13: Based on the physical status information, the memory allocator is used to classify, manage, and recycle memory modules with different keyword information.

[0065] The embodiment of the present invention binds the memory management unit inside the memory allocator and the memory module of the database through an established mapping relationship, so that the memory allocator and the upper system (such as the database system) with the corresponding memory module can obtain each other's actual information based on the mapping relationship, thereby classifying and managing and recycling memories of different memory levels and types, alleviating the adaptive memory management of the upper database system under different service pressures, and ensuring the timely release of physical real memory when the database service pressure increases. For cloud-native database scenarios with decentralized kernel memory management, according to actual memory management needs, only the memory modules that need to be adjusted according to the service are bound to the memory management unit, and the memory that needs to be adjusted is classified, managed and recycled, which can ensure the availability of the instance while improving the efficiency of memory management and recycling.

[0066] In an embodiment of the invention, the keyword information includes a first keyword (referred to as key in the embodiment of the invention) and a second keyword (referred to as classification key in the embodiment of the invention). The first keyword is used to identify the memory level type to which the memory module belongs, and the second keyword is used to identify the usage category of the memory module. The step of establishing a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator includes: establishing a first mapping relationship between the first keyword key and the second keyword classification key, and establishing a second mapping relationship between the second keyword key and the memory management unit. Then, in the process of the memory module requesting memory through the memory management unit, the physical status information of the memory management unit is retrieved and recorded based on the first mapping relationship and the second mapping relationship.

[0067] See Figure 4 and Figure 5 In an embodiment of the present invention, the memory allocator includes a Jemalloc memory allocator. The Jemalloc memory allocator is a general malloc implementation that focuses on reducing memory fragmentation and improving memory allocation efficiency in high-concurrency scenarios. The Jemalloc memory allocator divides the memory allocation granularity into three categories: Small, Large, and Huge. In the scenario of large memory allocation, the memory fragmentation of the Jemalloc memory allocator is less than that of other types of memory allocators currently available, such as less than the memory fragmentation of the tcmalloc memory allocator. The Jemalloc memory allocator has efficient memory allocation and recovery, improves performance in single-threaded or multi-threaded scenarios, can reduce memory fragmentation, including internal fragmentation and external fragmentation, and improves the effective utilization of memory. The memory management unit is the concurrent allocation unit (arena) of the Jemalloc memory allocator. The embodiment of the present invention saves the mapping relationship between keys and classification keys by adding a two-level hash map in the Jemalloc memory allocator (see Figure 4 , Figure 4The memory allocation module column is the first keyword key, the memory classification column is the second keyword classification key), and the mapping relationship between the classification key and the concurrent allocation unit (arena) of Jemalloc (see Figure 5 ) to manage the allocation of concurrent allocation units (arenas) according to database semantics. As system software, the database code itself is divided into many modules according to function. Different modules define a memory key in the program, such as key_memory_filesort_buffer_sort_keys, indicating that the memory allocation is used for sorting. Some are created for temporary tables. The memory lifecycle is managed uniformly based on the second keyword classification key. A bipolar hash map is used to establish a first mapping relationship between the first keyword key and the second keyword classification key. The key is used as the key and the classification key as the value to establish a key-value mapping relationship and store it in Jemalloc. When establishing a second mapping relationship between the second keyword classification key and the concurrent allocation unit (arena), the classification key is used as the key and the pointer to the array of one or more concurrent allocation units (arena) as the value to establish a key-value mapping relationship and store it in Jemalloc. Based on the first and second mapping relationships, the upper-level database can track the pointer to the array of one or more concurrent allocation units (arena) to retrieve the physical status information of each concurrent allocation unit (arena).

[0068] like Figure 5 As shown, one or more memory modules of different categories are mapped to one or more concurrent allocation units (arena), such as the data buffer pool module is mapped to two concurrent allocation units (arena), the additional memory pool and log memory cache are mapped to one concurrent allocation unit (arena), and different thread level caches are mapped to one concurrent allocation unit (arena). Figure 6 As shown, the upper-level database uses the thread cache to request a new region small memory block from the concurrent allocation unit (arena). When the concurrent allocation unit (arena) requests new memory from the system, the database tracks the memory based on the first mapping relationship and the second mapping relationship when allocating memory. The memory allocator returns the relevant physical status information of the concurrent allocation unit (arena) to the upper-level database, where the relevant physical status information includes memory mapping size mmap size, region filled, memory extension size extent size, memory extension type extent type, specification type bintype, specification size bin size, etc. The database collects and stores this information for subsequent analysis and management.

[0069] Wherein, the first keyword is placed in the index block header of the memory module. Then, in step S12, during the process in which the memory module requests memory through the memory management unit, the step of retrieving and recording the physical state information of the memory management unit based on the first mapping relationship and the second mapping relationship includes:

[0070] In the process of the memory module requesting memory through the memory management unit, the first keyword placed in the index block header is intercepted, and the physical state information is retrieved and recorded based on the first keyword based on the first mapping relationship and the second mapping relationship, such as by using a hook function to retrieve and record the physical state information based on the first keyword key of the database semantics based on the first mapping relationship and the second mapping relationship, thereby counting, limiting and managing the use and release of memory, such as Figure 4 In the process, after intercepting the relevant physical status information of the memory management unit, the thread-level memory usage is counted through the thread-level memory classification statistics mapping table to determine the memory usage of memory modules of different usage types. For example, the Jemalloc memory allocator as the callee is embedded in the bottom layer of the database application, that is, the database layer is the caller at the upper layer, and the Jemalloc memory allocator is at the lower layer. The Jemalloc memory allocator only contains concurrent allocation unit (arena) information (that is, physical status information, such as bin type, bin size, etc.). The existing database layer cannot retrieve (arena) information. The embodiment of the present invention adds a two-level mapping relationship to the interface for interaction between the database and the memory allocator. Therefore, the upper-layer database can call and view the concurrent allocation units (arena) corresponding to the memory modules with different classification keys through the interface. Figure 5 In MySQL, the relational database management system classifies the global-level memory by different usage categories, mapping the key to the corresponding arena in the Jemalloc allocator, and the session-level memory by different usage categories, mapping the key to the concurrent allocation unit (arena), thereby managing and recycling different memories in a hierarchical and classified manner.

[0071] Therefore, an embodiment of the present invention adds a database semantic keyword key to the interface for memory allocation and release interaction between the database and the memory allocator and records it in the allocated memory. The following code explains the use of the memory allocation and release interface, the addition of the keyword key part storage my_memory_header, and the use of the hook function MANAGED_MEMORY_CALL to count, limit, and manage memory usage and release based on the database semantic key.

[0072] / / Allocate memory header structure.

[0073] struct my_memory_header{

[0074] PSI_memory_key m_key; / / Module keyword key.

[0075] uint m_magic; / / Magic number used to determine if memory may be corrupted.

[0076] size_t m_size; / / The actual memory allocation size.

[0077] thread*m_owner; / / The address of the thread to which the memory belongs.

[0078] };

[0079] typedef struct my_memory_header my_memory_header;

[0080] / / Allocate memory wrapper function.

[0081] void*my_malloc(memory_key key,size_t size,myf flags){ ......

[0083] / / Add HOOK function to count and increase the module memory usage.

[0084] MANAGED_MEMORY_CALL(memory_alloc)(key,size)

[0085] / / Call the function that actually allocates memory.

[0086] my_raw_malloc(raw_size,flags) ......

[0088] }

[0089] / / Release memory wrapper function.

[0090] void my_free(void*ptr){ ......

[0092] / / Call the function that actually releases the memory.

[0093] my_raw_free(mh);

[0094] / / Add HOOK function to reduce the module memory usage.

[0095] MANAGED_MEMORY_CALL(memory_free)(mh->m_key,mh->m_size);

[0096] }.

[0097] Figure 4 The thread-level memory classification statistics mapping adds a hook function to the database to count the memory usage of different memory modules. Based on the usage size, the database uses the memory allocator to classify and manage different categories of memory modules with different strategies for recycling.

[0098] Due to the single transparent use of the memory allocator and the single memory reclamation strategy, when the database service pressure increases, the actual physical memory usage cannot be released in time, and the setting of a strong reclamation mechanism has a significant impact on the service. To better manage different memories and avoid the performance impact of a uniformly set management strategy, step S13 of the embodiment of the present invention uses the memory allocator to classify and manage and reclaim memory modules with different keyword information, including: using the memory allocator to classify and manage and reclaim memory modules with different keyword information using different memory management strategies, thereby reducing unnecessary allocation and release. For example, when the upper-level database calls the program's free to release a certain amount of memory, the memory may not be truly released due to the upper-level database reusing this part of the memory. After retrieving the physical status information of each memory management unit of the memory allocator using the database semantic keyword information, the upper-level database can determine that this part of the memory can actually be released. It can then notify the bin of the concurrent allocation unit (arena) corresponding to this part of the memory in the memory allocator to release the actual physical memory. In cases where the upper layer can reuse the memory but does not, the memory allocator is used to appropriately reuse the memory according to the memory usage of the corresponding memory module to improve performance.

[0099] like Figure 7When the memory management method provided by an embodiment of the present invention is adopted for the cloud-native relational database PolarDB MySQL, the overall configuration strategy architecture diagram of the database is shown. The Jemalloc memory allocator is embedded in the database, and the Jemalloc interface is modified by establishing a bipolar hash map so that the arena of the Jemalloc memory allocator can be bound to the memory module of the cloud-native relational database PolarDB MySQL. The upper-layer cloud-native database retrieves the relevant information of the concurrent allocation unit (arena) in the Jemalloc memory allocator, and uses the management strategy of the Jemalloc memory allocator to classify, manage and recycle the corresponding memory modules. Users interact with the database through the memory management user interface. The overall configuration strategy of the embodiment of the present invention is mainly watermark management, wherein, when the memory level type to which the memory module belongs includes instance level and / or session level, after step S12 retrieves and records the physical status information of the memory management unit, it also determines the memory usage of the memory modules with different keyword information based on the physical status information. Then, step S13 uses the memory allocator to classify, manage and recycle the memory modules with different keyword information based on the physical status information. Specifically, the steps include: using the memory allocator to adopt a watermark management strategy, and using the memory management unit to classify, manage and recycle the corresponding memory modules based on the keyword information. The watermark management strategy is a strategy for classifying, managing and reclaiming the memory modules when the memory usage exceeds the preset usage warning value. Figure 8 In the example, the water level is the memory usage. Figure 8 The 6GB, 10GB, and 11.5GB set in the memory are specific watermarks. If the memory usage of the corresponding memory module exceeds the corresponding watermark, the memory allocator 30 adopts the watermark management strategy, and the memory management unit manages the corresponding memory module based on the second keyword classification key. That is, the memory management unit tracks the memory module that needs to be managed based on the first mapping relationship and the second mapping relationship established based on the second keyword classification key. The memory manager can use the memory management unit to take strategies such as interrupting SQL execution, setting the maximum number of recycling threads, starting recycling threads, and default settings to classify and manage and recycle memory modules. It should be noted that the management strategy adopted by the memory manager is not limited to Figure 9 The displayed policy methods may include blocking new connections, interrupting large queries, clearing caches, etc.

[0100] When the level type to which the memory module belongs includes one or more of the instance level, session level and structured query language SQL level, step S13 includes the following steps: using the memory allocator to classify and manage and recycle memory modules with different keyword information according to the physical state information: using the memory allocator to adopt a periodic cleanup strategy, and using the memory management unit to periodically clean up the corresponding memory modules based on the keyword information to release physical memory. The periodic cleanup strategy is a strategy for cleaning up the memory modules according to a pre-set recycling cycle. Specifically, the memory management unit cleans up the corresponding memory modules based on the second keyword classification key, and according to the first mapping relationship and the second mapping relationship, tracks the memory modules that need to be periodically cleaned up to release physical memory. Figure 9 In the , session-level memory can be recycled after the session ends. The upper layer uses the bins and regions of the memory allocator arena to release and clean up the real physical memory based on the statistical memory usage of each thread and the mapping relationship between each thread and the arena. The instance level can set a regular recycling time for regular recycling.

[0101] See Figure 10 The second embodiment of the present invention further provides a memory management system, including a keyword addition module 10, a mapping module 20 and a recycling management module 40. The keyword addition module 10 is used to preset keyword information in the memory module, and the keyword information is used to identify the memory level type and usage category to which the memory module belongs. The mapping module 20 is used to establish a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator in the memory allocator, so that in the process of the memory module requesting memory through the memory management unit, the physical status information of the memory management unit is retrieved and recorded based on the mapping relationship. The recycling management module 40 is used to use the memory allocator to classify, manage and recycle memory modules with different keyword information according to the physical status information.

[0102] Among them, the keyword information includes a first keyword and a second keyword, the first keyword is used to identify the memory level type to which the memory module belongs, and the second keyword is used to identify the usage category of the memory module, wherein the step of the mapping module 20 establishing a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator includes: establishing a first mapping relationship between the first keyword and the second keyword, and establishing a second mapping relationship between the second keyword and the memory management unit, then in the process of the memory module requesting memory through the memory management unit, based on the first mapping relationship and the second mapping relationship, the physical status information of the memory management unit is retrieved and recorded.

[0103] The first keyword is placed in the index block header of the memory module. The memory management system provided by the embodiment of the present invention also includes a data interception module 50. The data interception module 50 is used to intercept the first keyword placed in the index block header during the process of the memory module requesting memory through the memory management unit, and retrieve and record the physical status information based on the first keyword based on the first mapping relationship and the second mapping relationship.

[0104] In an embodiment of the present invention, the memory allocator includes a Jemalloc memory allocator, and the memory management unit is a concurrent allocation unit (arena) of the Jemalloc memory allocator. In the embodiment of the present invention, under the premise of deeply binding memory modules of different levels and different life cycles to the memory allocator, the semantics of the database instance, library, session and SQL statement levels are transparently transmitted to the more mature Jemalloc memory allocator through the keyword information marked during database memory allocation, thereby reducing the cost of transformation. Although the embodiment of the present invention can also use any open source malloc, such as the tcmalloc allocator and the ptmalloc allocator, the tcmalloc allocator and the ptmalloc allocator are not as suitable as the Jemalloc memory allocator. Compared with other memory allocators, the Jemalloc memory allocator has high memory allocation performance in multi-threaded scenarios and can reduce memory fragmentation. Therefore, the Jemalloc memory allocator is preferably used to implement memory management, thereby using the flexible and powerful memory management method of the Jemalloc memory allocator to classify, manage and recycle different memory modules, thereby greatly alleviating the adaptive memory management of the upper-level database system under different service pressures.

[0105] When the memory level type to which the memory module belongs includes instance level and / or session level, the memory management system retrieves and records the physical status information of the concurrent allocation unit (arena), and also determines the memory usage of the memory modules with different keyword information based on the physical status information. Then, the recycling management module 40 uses the memory allocator to classify and manage and recycle the memory modules with different keyword information based on the physical status information. The steps specifically include: using the memory allocator to adopt a watermark management strategy, and using the memory management unit to classify and manage and recycle the corresponding memory modules based on the keyword information. The watermark management strategy is a strategy for classifying and managing and reclaiming memory modules when the memory usage exceeds a preset usage warning value. The watermark is the memory usage rate. Figure 8The 6GB, 10GB, and 11.5GB set in the memory management system are specific watermarks. If the memory usage of the corresponding memory module exceeds the corresponding watermark, the memory management system can use the memory allocator 30 to adopt the watermark management strategy, and manage the corresponding memory module based on the second keyword classification key through the memory management unit. That is, the memory management unit tracks the memory module that needs to be managed based on the first mapping relationship and the second mapping relationship established based on the second keyword classification key. The memory manager can use the memory management unit to take strategies such as interrupting SQL execution, setting the maximum number of recycling threads, starting recycling threads, and default settings to classify, manage, and recycle memory modules. It should be noted that the management strategy adopted by the memory manager is not limited to Figure 9 The displayed policy methods may include blocking new connections, interrupting large queries, clearing caches, etc.

[0106] When the level type to which the memory module belongs includes one or more of the instance level, session level and structured query language SQL level, the recycling management module 40 uses the memory allocator to classify, manage and recycle memory modules with different keyword information according to the physical state information, and the step also includes: using the memory allocator to adopt a periodic cleaning strategy, and using the memory management unit to periodically clean up the corresponding memory modules based on the keyword information to release physical memory. The periodic cleaning strategy is a strategy for cleaning up the memory modules according to a pre-set recycling cycle. Specifically, the memory management unit cleans up the corresponding memory modules based on the second keyword classification key, and according to the first mapping relationship and the second mapping relationship, tracks the memory modules that need to be regularly cleaned up to release physical memory. Figure 9 In the , session-level memory can be recycled after the session ends. The upper-level memory management system uses the memory usage statistics of each thread and the mapping relationship between each thread and the arena to release and clean up the real physical memory using the bin and region of the memory allocator arena. The instance level can set a regular recycling time for regular recycling.

[0107] Therefore, when the memory management system provided by the embodiment of the present invention is applied to a database management system, the database management system is based on the Jemalloc memory allocator, so that its internal concurrent allocation unit (arena) can be bound to the memory usage of the database semantics, that is, through two-level hash mapping management, different management strategies are performed according to different database memory classifications. The concurrent allocation unit (arena) of the Jemalloc memory allocator is only classified according to the memory block size range, which is more conducive to the management and optimization of the database's real physical memory.

[0108] The embodiment of the present invention adopts the premise of deep binding between database memory and memory allocator, and transparently transmits the semantics of database instance, library, session and SQL statement levels to Jemalloc memory allocator through keyword information marked in memory module when database memory is allocated. With the help of Jemalloc's flexible and powerful memory management method, different memory modules are classified, managed and recycled, so as to alleviate the adaptive memory management of the upper-level database system under different service pressures.

[0109] The third embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method steps described in the first embodiment of the present invention when executing the computer program.

[0110] The fourth embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method steps described in the first embodiment of the present invention are implemented.

[0111] The fifth embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method steps described in the first embodiment of the present invention are implemented.

[0112] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A memory management method, comprising: Presetting keyword information in the memory module, the keyword information is used to identify a memory level type and a usage category to which the memory module belongs, the memory level type including instance level, database level, session level, or SQL level, the usage category including sort buffer, temporary table space, connection cache, or read data cache, and the memory module is a memory module of a cloud native database; A mapping relationship between the keyword information and a memory management unit corresponding to the memory allocator (30) is established in the memory allocator (30), so that in the process of the memory module requesting memory through the memory management unit, based on the mapping relationship, the physical status information of the memory management unit is retrieved and recorded, the physical status information of the memory management unit including the amount of physical memory actually allocated to each memory module and the currently adopted memory management strategy, and the memory management unit is a concurrent allocation unit of the memory allocator; According to the physical state information, the memory modules with different keyword information are classified, managed, and recycled using the memory allocator (30), specifically comprising: for the memory modules with different keyword information, using the memory allocator (30) to classify, manage, and recycle using different memory management strategies; The keyword information includes a first keyword and a second keyword; the first keyword is used to identify the memory level type to which the memory module belongs, and the second keyword is used to identify the usage category of the memory module, wherein the step of establishing a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator includes: A first mapping relationship between the first keyword and the second keyword is established, and a second mapping relationship between the second keyword and the memory management unit is established. Then, in the process of the memory module requesting memory through the memory management unit, the physical status information of the memory management unit is retrieved and recorded based on the first mapping relationship and the second mapping relationship. The second mapping relationship between the second keyword and the memory management unit is established by establishing a mapping relationship between the second keyword and a pointer to an array of one or more concurrent allocation units.

2. The method according to claim 1, wherein The first keyword is placed in the index block header of the memory module. Then, in a process in which the memory module requests memory through the memory management unit, the step of retrieving and recording the physical state information of the memory management unit based on the first mapping relationship and the second mapping relationship includes: During the process of the memory module requesting memory through the memory management unit, the first keyword placed in the index block header is intercepted, and the physical status information is retrieved and recorded based on the first keyword and the first mapping relationship and the second mapping relationship.

3. The method according to claim 1, wherein The memory level type to which the memory module belongs includes instance level and / or session level. After retrieving and recording the physical state information of the memory management unit, the method further includes: determining the memory usage of the memory modules having different keyword information according to the physical status information; Then, according to the physical status information, the steps of using the memory allocator (30) to classify, manage and recycle the memory modules with different keyword information include: The memory allocator (30) is used to adopt a watermark management strategy, and the memory management unit is used to classify and manage and recycle the corresponding memory modules based on the keyword information. The watermark management strategy is a strategy for classifying and managing and reclaiming the memory modules when the memory usage exceeds a preset usage warning value.

4. The method according to claim 1, wherein the step of using the memory allocator (30) to classify, manage and recycle the memory modules having different keyword information according to the physical status information further comprises: The memory allocator (30) is used to adopt a periodic cleaning strategy, and the memory management unit periodically cleans the corresponding memory module based on the keyword information to release physical memory. The periodic cleaning strategy is a strategy for cleaning the memory module according to a preset recycling cycle.

5. A memory management system comprising: A keyword adding module (10) is used to preset keyword information in a memory module, wherein the keyword information is used to identify a memory level type and a usage category to which the memory module belongs, wherein the memory level type includes an instance level, a database level, a session level, or an SQL level, and the usage category includes a sort buffer, a temporary table space, a connection cache, or a read data cache, and the memory module is a memory module of a cloud native database; A mapping module (20) is used to establish a mapping relationship between the keyword information and a memory management unit corresponding to the memory allocator (30), so as to retrieve and record physical status information of the memory management unit based on the mapping relationship when the memory module requests memory through the memory management unit, wherein the physical status information of the memory management unit includes the amount of physical memory actually allocated to each memory module and the memory management strategy currently adopted, and the memory management unit is a concurrent allocation unit of the memory allocator; A recycling management module (40) is used to classify and manage the memory modules with different keyword information using the memory allocator (30) according to the physical state information, specifically comprising: using the memory allocator (30) to classify and manage and recycle the memory modules with different keyword information using different memory management strategies; The keyword information includes a first keyword and a second keyword; the first keyword is used to identify the memory level type to which the memory module belongs, and the second keyword is used to identify the usage category of the memory module, wherein the step of establishing a mapping relationship between the keyword information and the memory management unit corresponding to the memory allocator includes: A first mapping relationship between the first keyword and the second keyword is established, and a second mapping relationship between the second keyword and the memory management unit is established. Then, in the process of the memory module requesting memory through the memory management unit, the physical status information of the memory management unit is retrieved and recorded based on the first mapping relationship and the second mapping relationship. The second mapping relationship between the second keyword and the memory management unit is established by establishing a mapping relationship between the second keyword and a pointer to an array of one or more concurrent allocation units.

6. The system according to claim 5, wherein: The first keyword is placed in an index block header of the memory module, and the system further includes: A data interception module is used to intercept the first keyword placed in the index block header during the process of the memory module requesting memory through the memory management unit, and to retrieve and record the physical status information based on the first keyword based on the first mapping relationship and the second mapping relationship.

7. The system according to any one of claims 5 to 6, wherein: The memory allocator (30) includes a Jemalloc memory allocator, and the memory management unit is a concurrent allocation unit (arena) of the Jemalloc memory allocator.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, wherein The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images