Database system memory management method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a database system memory management method and device, electronic equipment and a storage medium. The method comprises the steps that the total memory value occupied by table caches of level db instances corresponding to all fragments is obtained regularly; judging whether the total memory value exceeds a total memory threshold value occupied by a preset table cache or not; and under the condition that the total memory value is judged to exceed a preset total table cache occupied memory threshold value, performing memory data cleaning based on the table cache of the level db instance corresponding to each fragment until the total memory value does not exceed the preset total table cache occupied memory threshold value. According to the technical scheme, the total threshold value of the occupied memory is set for the table caches of all the level db instances in the database system, and when the total value of the memory occupied by the table caches of the level db instances corresponding to all the fragments exceeds the total threshold value, memory data cleaning is carried out. The problem that in the prior art, due to the fact that the number of generated level db instances of a database system is increased, the memory occupied by a table cache is continuously increased, and the level db instances are killed by an operation system is effectively solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of database management, and in particular to a method, device, electronic device and storage medium for memory management of a database system. Background Art

[0002] The database system that uses leveldb (level data base, lightweight database) as the storage engine provides storage and real-time access to massive structured and semi-structured data. The database system divides the data table into multiple shards according to preset rules and allocates a leveldb instance to each shard for storage and read and write services.

[0003] The module that mainly occupies memory in the above database system is the leveldb instance, and the modules that mainly occupy memory in the leveldb instance are the block cache and the table cache. In the prior art, in order to alleviate the situation where the database system occupies too much memory, the database system uses an instance-shared memory management method for the block cache of the leveldb instance, and limits the memory occupied by the block cache by limiting the size of this block cache; for the table cache of the leveldb instance, the database system can control the maximum number of data file handles of the data files corresponding to the shards that can be cached by the table cache of each leveldb instance through the option max_open_files, and then control the number of metadata of the data files corresponding to the data file handles loaded into the memory, so as to manage the memory occupied by the table cache of the leveldb instance.

[0004] However, with the existing management method for the memory occupied by the table cache of the leveldb instance, as the user storage data increases, the shards managed by the database system process also continue to grow, so the number of leveldb instances generated also increases, causing the memory occupied by the table cache of the database system to continue to grow until the database system's memory is exhausted and killed by the operating system, thereby affecting the availability of the database system service. Summary of the invention

[0005] In view of this, in order to alleviate the problem that as the user stores more data, the memory occupied by the table cache of the database system continues to grow until the database system's memory is exhausted and killed by the operating system, thereby affecting the availability of the database system service, the embodiments of the present application provide a method, device, electronic device and storage medium for database system memory management.

[0006] In a first aspect, an embodiment of the present application provides a method for memory management of a database system, the method comprising:

[0007] Regularly obtain the total memory occupied by the table cache of the leveldb instance corresponding to all shards;

[0008] Determine whether the total memory value exceeds a preset table cache occupied memory total threshold;

[0009] When it is determined that the total memory value exceeds the preset total memory threshold occupied by the table cache, memory data is cleaned up based on the table cache of the leveldb instance corresponding to each of the shards until the total memory value does not exceed the preset total memory threshold occupied by the table cache.

[0010] Optionally, as in the aforementioned method, the periodic acquisition of the total memory value occupied by the table cache of the leveldb instance corresponding to all shards includes:

[0011] Periodically obtain the data file handle of each shard cached in the table cache of the leveldb instance of each shard;

[0012] Querying in memory the memory occupied value of the metadata of the data files of the shards corresponding to each of the data file handles;

[0013] The total memory value is determined based on each of the occupied memory values.

[0014] Optionally, as in the aforementioned method, the memory data cleaning based on the table cache of the leveldb instance corresponding to each of the shards includes:

[0015] Obtain the data weight value corresponding to each of the shards, and the number of accesses to the table cache of the leveldb instance corresponding to each of the shards within a preset time period;

[0016] Determine a target shard from the plurality of shards based on the data weight values ​​and the number of accesses corresponding to each shard;

[0017] Clean up data of the data file handle of the target shard cached in the table cache of the leveldb instance corresponding to the target shard;

[0018] The metadata of the data file of the target shard corresponding to the data file handle is cleaned from the memory.

[0019] Optionally, as in the aforementioned method, determining a target shard from the plurality of shards based on the data weight values ​​and the number of accesses corresponding to each shard includes:

[0020] For each of the shards, determining a score value of the shard according to the data weight value and the number of access times;

[0021] The target shard is determined from the plurality of shards based on the score values ​​corresponding to the shards.

[0022] Optionally, as in the aforementioned method, determining the target shard from the plurality of shards based on the score values ​​corresponding to the shards includes:

[0023] Selecting a first score value having the smallest score value and exceeding a first preset score value from the plurality of score values;

[0024] Determining whether there are multiple first score values;

[0025] If it is determined that the first score value is not multiple, determining the slice corresponding to the first score value as the target slice;

[0026] When it is determined that there are multiple first score values, selecting a second score value having the smallest data weight value corresponding to the slice of the first score value from the multiple first score values;

[0027] determining whether the second score value is multiple;

[0028] If it is determined that the second score value is not multiple, determine the slice corresponding to the second score value as the target slice;

[0029] When it is determined that there are multiple second score values, one second score value is randomly selected from the multiple second score values, and the shard corresponding to the selected second score value is determined as the target shard.

[0030] Optionally, as in the aforementioned method, the method further comprises:

[0031] The score value corresponding to the target slice is reset to a second preset score value.

[0032] Optionally, as in the aforementioned method, determining a target shard from the plurality of shards based on the data weight values ​​and the number of accesses corresponding to each shard includes:

[0033] Selecting the first shard with the smallest data weight value and the smallest number of access times from each of the shards;

[0034] Determine whether there are multiple first fragments;

[0035] If it is determined that there is not more than one first shard, determining the first shard as the target shard;

[0036] When it is determined that there are multiple first shards, one first shard is randomly selected from the multiple first shards, and the selected first shard is determined as the target shard.

[0037] In a second aspect, an embodiment of the present application provides a device for memory management of a database system, the device comprising:

[0038] The acquisition module is used to periodically obtain the total memory occupied by the table cache of the leveldb instance corresponding to all shards;

[0039] A determination module, used to determine whether the total memory value exceeds a preset table cache occupied total memory threshold;

[0040] A data cleaning module is used to clean up the table cache of the leveldb instance corresponding to the shard when it is determined that the total memory value exceeds the total memory occupied by the preset table cache, until the total memory value does not exceed the total memory occupied by the preset table cache.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, which includes: a processor and a memory, the processor is used to execute a database system memory management program stored in the memory to implement the above-mentioned database system memory management method.

[0042] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned database system memory management method.

[0043] The method, device, electronic device and storage medium for database system memory management provided by the embodiment of the present application include: regularly obtaining the total memory value occupied by the table cache of the leveldb instance corresponding to all shards; determining whether the total memory value exceeds the preset total memory threshold occupied by the table cache; when determining that the total memory value exceeds the preset total memory threshold occupied by the table cache, memory data is cleaned up based on the table cache of the leveldb instance corresponding to each shard until the total memory value does not exceed the preset total memory threshold occupied by the table cache. In the above technical scheme, a total threshold of the occupied memory, i.e., the preset total memory threshold occupied by the table cache of all leveldb instances in the database system is set, and memory data is cleaned up when the total memory value occupied by the table cache of the leveldb instance corresponding to all shards exceeds this total threshold, so that when the database system is continuously running, the total memory value occupied by the table cache of the leveldb instance will not exceed the preset total threshold as the number of leveldb instances generated increases, effectively alleviating the problem in the prior art that the memory occupied by the table cache continues to grow due to the increase in the number of leveldb instances generated by the database system, until the memory of the database system is exhausted and killed by the operating system, thereby improving the availability of the database system service. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A schematic diagram of an application environment of a method for memory management of a database system provided in an embodiment of the present application;

[0047] Figure 2 A flowchart of a method for memory management of a database system provided in an embodiment of the present application;

[0048] Figure 3 A flowchart of another method for memory management of a database system provided in an embodiment of the present application;

[0049] Figure 4 A flowchart of another method for memory management of a database system provided in an embodiment of the present application;

[0050] Figure 5A flowchart of another method for memory management of a database system provided in an embodiment of the present application;

[0051] Figure 6 A flowchart of another method for memory management of a database system provided in an embodiment of the present application;

[0052] Figure 7 A block diagram of an embodiment of a database system memory management device provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0057] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0060] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present application.

[0061] The present application provides a method for managing database system memory, which can be applied in the following situations: Figure 1 In an application environment, a server communicates with a client, the server provides a database system memory management service, and the client sends a database system memory management task to the server. The client includes but is not limited to a PDA, a desktop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud computer device, a personal digital assistant (PDA), and other devices. The computer device corresponding to the server can be implemented by an independent server or a server cluster consisting of multiple servers.

[0062] See also Figure 2 , is a flow chart of a method for memory management of a database system provided in an embodiment of the present application, wherein the method for memory management of a database system is applied to Figure 1 The server in the server is specifically a database system in the server to manage the memory corresponding to the database system. In this embodiment, the database system is a NoSQL (Not Only Structured Query Language, non-relational database) distributed database system that uses leveldb as a storage engine and can generate multiple leveldb instances.

[0063] like Figure 2 As shown, the database system memory management method may include the following steps:

[0064] Step 101, regularly obtain the total memory occupied by the table cache of the leveldb instance corresponding to all shards;

[0065] The above-mentioned sharding usually involves slicing all the data tables created in the database system according to preset rules to obtain multiple smaller data fragments, each of which is called a shard. Specifically, key-based sharding is a common sharding method, which divides the data table into rows, and each shard contains a portion of rows, and these rows have the same key value range. In addition to sharding each data table according to the preset rule of key, each data table can also be sharded according to preset rules such as range sharding, hash sharding, geographic location sharding, consistent hash sharding, random sharding, etc., which will not be described one by one here. Among them, the preset rules for data table sharding are not limited to the several methods mentioned above.

[0066] In order to achieve data isolation and parallel processing and improve the overall performance and reliability of the database system, after each data table is divided into multiple shards, an independent leveldb instance is assigned to each shard to ensure that the data of different shards are isolated from each other and to achieve parallel processing of data. Among them, each shard stores a part of the data of the collection, which is divided into multiple data blocks, each data block is stored in a data file. In this embodiment, the table cache of each leveldb instance can be used to cache the data file handle of the data file corresponding to the shard, and the leveldb instance loads the metadata of the data file of the shard corresponding to the data file handle cached in the table cache into the memory for storage.

[0067] The above total memory value can be understood as the sum of the memory occupied by the metadata of the sharded data files corresponding to the data file handles cached in the table cache of all leveldb instances opened by the database system at the current time corresponding to the regular period.

[0068] The specific process of determining the above total memory value is: regularly obtain the data file handles of the shards cached in the table cache of the leveldb instance of each shard; query the occupied memory value of the metadata of the data file of the shard corresponding to each data file handle in the memory; determine the total memory value based on each occupied memory value.

[0069] For example, a database system process opens 4000 leveldb instances at 13:00 in the afternoon. The table cache of each leveldb instance caches 100 sharded data file handles. The metadata size of the sharded data file corresponding to each data file handle is 0.5MB. Then the total memory value in memory is 200000MB. If the database system process opens 8000 leveldb instances at 13:05 in the afternoon, and the table cache of each leveldb instance caches 100 sharded data file handles, the metadata size of the sharded data file corresponding to each data file handle is 0.5MB, then the total memory value in memory is 400000MB.

[0070] As can be seen from the above, the above period is 5 minutes, that is, the total value of the memory occupied by the table cache of the leveldb instance corresponding to all shards is obtained once every 5 minutes. In actual application, the periodic duration can be set according to actual needs and is not limited here.

[0071] Step 102, determining whether the total memory value exceeds a preset table cache occupied total memory threshold;

[0072] In this embodiment, the table caches of all leveldb instances in the database system set a total threshold of the occupied memory, that is, the above-mentioned preset total threshold of the memory occupied by the table cache. This threshold is related to the available memory of the database system and is not limited here.

[0073] Step 103, when it is determined that the total memory value exceeds the preset table cache memory occupied total threshold, memory data is cleaned up based on the table cache of the leveldb instance corresponding to each shard until the total memory value does not exceed the preset table cache memory occupied total threshold.

[0074] The total memory value exceeds the preset total memory occupied by the table cache threshold, indicating that the metadata of the data files of the shards corresponding to the data file handles cached in the table caches of all leveldb instances opened by the database system occupy more memory in the memory. In this embodiment, in order to effectively avoid the problem that the memory occupied by the table cache continues to grow due to the increase in the number of leveldb instances generated by the database system until the memory of the database system is exhausted and killed by the operating system, it is necessary to clean up the metadata of the data files corresponding to the shards stored in the memory, so that when the database system continues to run, the total memory occupied by the table cache of the leveldb instance will not exceed the preset total memory occupied by the table cache threshold as the number of leveldb instances generated increases.

[0075] For example, if the total threshold of memory occupied by the preset table cache is set to 100GB, then as the number of leveldb instances generated by the database system increases and the memory occupied by the table cache continues to grow by more than 100GB, it is necessary to clean up the metadata of the data files corresponding to the shards stored in the memory, thereby reducing the total memory occupied by the table caches of the leveldb instances corresponding to all shards. The memory data cleaning will not be stopped until the total memory value is reduced to no more than 100GB.

[0076] Through the method in this embodiment, a total threshold of the memory occupied by the table caches of all leveldb instances in the database system, i.e., a preset total threshold of the memory occupied by the table caches, is set. When the total value of the memory occupied by the table caches of the leveldb instances corresponding to all shards exceeds this total threshold, data cleaning of the table caches is performed. As a result, when the database system continues to run, the total value of the memory occupied by the table caches of the leveldb instances will not exceed the preset total threshold as the number of leveldb instances generated increases. This effectively alleviates the problem in the prior art that the memory occupied by the table caches continues to grow due to the increase in the number of leveldb instances generated by the database system until the memory of the database system is exhausted and the system is killed by the operating system, thereby improving the availability of database system services.

[0077] like Figure 3 As shown, as an optional implementation, as in the aforementioned method, step 103 performs memory data cleaning based on the table cache of the leveldb instance corresponding to each shard, including the following steps:

[0078] Step 301, obtaining the data weight value corresponding to each shard, and the number of accesses to the table cache of the leveldb instance corresponding to each shard within a preset time period;

[0079] In order to minimize the impact of the data cleared from the memory on the performance of the database system, in this embodiment, the influence of each shard on the performance of the database system can be measured by two indicator data, namely the data weight value of the shard and the number of accesses to the table cache. Therefore, the metadata of the data files corresponding to the shard in the memory that has the smallest impact on the performance of the database system can be cleaned up, thereby achieving the management of the total size of the memory occupied by the database system without affecting the performance of the database system.

[0080] In one embodiment, a data weight value is set for each data table, so that the data weight value corresponding to the slice after the data table is cut is the data weight value of the data table. In another embodiment, a data weight value can be directly set for each slice.

[0081] The higher the above data weight value is, the less likely the metadata of the data file corresponding to this shard will be cleared. When the metadata is cleared, if the data of this shard is accessed again, its leveldb instance needs to reload the data of this shard from the hard disk into the memory, which will increase the request processing time. Therefore, for shards or tables that are frequently accessed and have a relatively uniform access key range, a larger data weight value can be set; and for shards or tables that are rarely accessed or have a relatively concentrated access key range, such as keys starting with dates and businesses concentrated on accessing data in the recent period, a smaller data weight value can be set. By setting this data weight value, the metadata of frequently accessed shards is more likely to reside in memory to minimize the impact on the performance of the database system.

[0082] In actual application, when accessing the data of a shard, it is necessary to first query the data file handle of the data file corresponding to the data from the table cache of the leveldb instance, then query the metadata corresponding to the data file handle from the memory, and finally access the data in the data file of this shard through the metadata query. Therefore, the number of accesses through the table cache of the leveldb instance can be used to directly represent whether the shard is frequently accessed within a preset time period.

[0083] The above preset time period is the time period closest to the regular current time. For example, if the regular current time is 13:00 p.m. and the preset time period is 4 minutes, then it is necessary to obtain the number of accesses to the table cache of the leveldb instance corresponding to each shard within the preset time period of 12:56 p.m. If the regular current time is 13:05 p.m., then it is necessary to obtain the number of accesses to the table cache of the leveldb instance corresponding to each shard within the preset time period of 13:01 p.m., where the length of the preset time period can be set according to actual needs and is not limited here.

[0084] In actual application, the number of visits can be counted using a sliding time window, a rolling window, a session window, a fixed time window, or a timestamp-based window. The method for counting the number of visits is not limited here.

[0085] Step 302, determining a target shard from multiple shards based on the data weight value and the number of accesses corresponding to each shard;

[0086] The target shard is the shard with the least impact on the performance of the database system, determined by the data weight value and the number of accesses.

[0087] Step 303, clearing the data file handle of the cached target shard in the table cache of the leveldb instance corresponding to the target shard;

[0088] Before cleaning the metadata of the data file of the target shard from the memory, you need to clean the data file handle of the data file of the target shard from the table cache of the leveldb instance corresponding to the target shard. This ensures that the cached data will be reloaded in subsequent operations, thereby avoiding data inconsistency problems and optimizing database system performance.

[0089] In summary, cleaning up the data file handles in the table cache first and then cleaning up the metadata in the memory can help ensure data consistency and improve database system performance.

[0090] Step 304: clear the metadata of the data file of the target shard corresponding to the data file handle from the memory.

[0091] By the method in this embodiment, when the total value of memory occupied by the table cache of the leveldb instance corresponding to all shards exceeds the preset total threshold of memory occupied by the table cache, a target shard with the smallest impact on the performance of the database system is selected from multiple shards based on the two data indicators of data weight value and number of accesses, so as to clean up the metadata of the target shard from the memory, so as to achieve the management of the total size of memory occupied by the database system without affecting the performance of the database system, so that when the database system continues to run, the total value of memory occupied by the table cache of the leveldb instance will not exceed the preset total threshold as the number of leveldb instances generated increases.

[0092] like Figure 4 As shown, as an optional implementation, as in the aforementioned method, the step 302 of determining the target shard from multiple shards based on the data weight value and the number of accesses corresponding to each shard includes the following steps:

[0093] Step 401, for each shard, determine the score value of the shard according to the data weight value and the number of access times;

[0094] The score value of each shard can be calculated using the following formula:

[0095] Score value = data weight value * number of visits.

[0096] Step 402: determine a target shard from multiple shards based on the score values ​​corresponding to each shard.

[0097] The above step 402 can be specifically performed as follows: Figure 5 This is done as shown:

[0098] Step 501, selecting a first score value having the smallest score value and exceeding a first preset score value from multiple score values;

[0099] Optionally, the first preset score value is 0. It is understandable that, considering the score calculation error, the first preset score value may be set to other values ​​other than 0. Of course, the first preset score value may also be other values, which are not specifically limited here.

[0100] Step 502, determining whether the first score value is multiple;

[0101] The purpose of the judgment is to determine whether there are multiple shards corresponding to the first score value. If it is determined that there are not multiple shards, step 503 is executed, that is, the shard corresponding to the first score value is determined as the target shard corresponding to the metadata cleanup; if it is determined that there are multiple first score values, step 504 is executed, that is, it is further necessary to determine the target shard for metadata cleanup through the data weight value.

[0102] Step 503, determining the slice corresponding to the first score value as the target slice;

[0103] Step 504, selecting a second score value having the smallest data weight value corresponding to the slice of the first score value from the multiple first score values;

[0104] Since the number of accesses has a certain temporal randomness, and the data weight value is an intuitive measurement indicator to measure the impact of shards on database system performance, therefore, when there are multiple first score values, in order to further determine the target shard with the least impact on database system performance from the shards corresponding to the multiple first score values, it is necessary to further determine the target shard through the data weight value.

[0105] The impact of each shard on the performance of the database system is measured by two indicators: the data weight value of the shard and the number of accesses to the table cache.

[0106] Step 505, determining whether the second score value is multiple;

[0107] The purpose of the determination is to determine whether there are multiple shards corresponding to the second score value. If it is determined that there are not multiple second score values, step 506 is executed, that is, the shard corresponding to the second score value is determined as the target shard for metadata cleanup. If it is determined that there are multiple second score values, step 507 is executed, indicating that the shards corresponding to the multiple second score values ​​have the same impact on the performance of the database system. Therefore, a shard corresponding to a second score value can be randomly selected from the multiple second score values ​​as the target shard for metadata cleanup.

[0108] Step 506, determining the slice corresponding to the second score value as the target slice;

[0109] Step 507: randomly select a second score value from the multiple second score values, and determine the shard corresponding to the selected second score value as the target shard.

[0110] In specific implementation, after the metadata of the data file corresponding to the target shard is cleaned up in memory, the score value of the target shard needs to be reset to a second preset score value, and the first preset score value may be 0. Of course, the first preset score value may also be other values, which are not specifically limited here. Since the metadata of the data file of the target shard is cleaned up from the memory, it means that the target shard has almost no effect on the performance of the database system. Therefore, the score value of the target shard is reset to ensure the consistency of the effect of the target shard on the performance of the database system.

[0111] Through the method in this embodiment, the score value of each shard can be determined using the data weight value and the number of access times, and then the target shard with the smallest impact on the performance of the database system can be accurately determined from multiple shards through the score value of each shard, so that the metadata of the data file corresponding to the target shard is cleaned up from the memory, thereby achieving management of the total memory size occupied by the database system without affecting the performance of the database system.

[0112] In addition to being able to determine the target shard based on the score values ​​corresponding to each shard, this embodiment also provides another method for determining the target shard based on the following example: Figure 6 As shown, as another optional implementation, as in the aforementioned method, the step 302 of determining the target shard from multiple shards based on the data weight value and the number of accesses corresponding to each shard includes the following steps:

[0113] Step 601, selecting the first shard with the smallest data weight value and the smallest number of accesses from each shard;

[0114] The first shard is the shard that has the least impact on the performance of the database system.

[0115] Step 602, determining whether there are multiple first fragments;

[0116] If it is determined that there are not multiple first fragments, step 603 is executed; if it is determined that there are multiple first fragments, step 604 is executed.

[0117] Step 603, determining the first shard as the target shard;

[0118] Step 604: randomly select a first shard from the multiple first shards, and determine the selected first shard as the target shard.

[0119] Through the method in this embodiment, the target shard with the least impact on the performance of the database system can be accurately determined from multiple shards through the data weight value and the number of access times, so that the metadata of the data file corresponding to the target shard can be cleaned up from the memory, thereby achieving the management of the total memory size occupied by the database system without affecting the performance of the database system.

[0120] In actual application, if a target shard is determined through steps 401 to 402, or through steps 601 to 604, and the metadata of the data file corresponding to the target shard is cleaned up, if the total memory value occupied by the table cache of the leveldb instance of all shards is still greater than the preset total memory threshold occupied by the table cache, then continue to select the next target shard according to the above steps 401 to 402, or through steps 601 to 604, and repeat the memory data cleaning until the total memory value does not exceed the preset total memory threshold occupied by the table cache, and stop the process of determining the target shard.

[0121] See also Figure 7 , is a block diagram of an embodiment of a database system memory management device provided in an embodiment of the present application. Figure 7 As shown, the device comprises:

[0122] The acquisition module 701 is used to periodically obtain the total memory occupied by the table cache of the leveldb instance corresponding to all shards;

[0123] A determination module 702 is used to determine whether the total memory value exceeds a preset table cache occupied total memory threshold;

[0124] The data cleaning module 703 is used to clean up the table cache of the leveldb instance corresponding to the shard when it is determined that the total memory value exceeds the preset table cache total memory threshold, until the total memory value does not exceed the preset table cache total memory threshold.

[0125] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0126] As an optional implementation, the acquisition module includes:

[0127] The first acquisition module is used to periodically obtain the data file handle of the shard cached in the table cache of the leveldb instance of each shard;

[0128] A query module is used to query the memory occupied value of the metadata of the data file of the shard corresponding to each data file handle in the memory;

[0129] The first determination module is used to determine the total memory value based on each occupied memory value.

[0130] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0131] As an optional implementation, the data cleaning module includes:

[0132] The second acquisition module is used to obtain the data weight value corresponding to each shard and the number of access times of the table cache of the leveldb instance corresponding to each shard within a preset time period;

[0133] A second determination module is used to determine a target shard from multiple shards based on the data weight value and the number of access times corresponding to each shard;

[0134] A first data cleaning module is used to clean up the data file handle of the target shard cached in the table cache of the leveldb instance corresponding to the target shard;

[0135] The second data cleaning module is used to clean the metadata of the data file of the target segment corresponding to the data file handle from the memory.

[0136] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0137] As an optional implementation, the second determining module includes:

[0138] A third determination module is used to determine the score value of each shard according to the data weight value and the number of access times;

[0139] The fourth determination module is used to determine a target shard from multiple shards based on the score values ​​corresponding to each shard.

[0140] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0141] As an optional implementation manner, the fourth determining module includes:

[0142] A first selection module, used to select a first score value having a minimum score value and a score value exceeding a first preset score value from a plurality of score values;

[0143] A first determination module, used to determine whether the first score value is multiple;

[0144] A fifth determination module, configured to determine the shard corresponding to the first score value as the target shard if it is determined that the first score value is not multiple;

[0145] A second selection module is used to select, when it is determined that there are multiple first score values, a second score value with the smallest data weight value corresponding to the slice of the first score value from the multiple first score values;

[0146] A second determination module is used to determine whether the second score value is multiple;

[0147] A sixth determination module, configured to determine the slice corresponding to the second score value as the target slice when it is determined that the second score value is not multiple;

[0148] The seventh determination module is used to randomly select a second score value from the multiple second score values ​​when it is determined that there are multiple second score values, and determine the shard corresponding to the selected second score value as the target shard.

[0149] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0150] As an optional implementation, the above device further includes:

[0151] The reset module is used to reset the score value corresponding to the target slice to a second preset score value.

[0152] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0153] As an optional implementation manner, the fourth determining module includes:

[0154] The third selection module is used to select the first shard with the smallest data weight value and the smallest number of accesses from each shard;

[0155] A third determination module is used to determine whether there are multiple first fragments;

[0156] an eighth determination module, configured to determine the first fragment as a target fragment if it is determined that there are not multiple first fragments;

[0157] The ninth determination module is used to randomly select a first slice from the multiple first slices when it is determined that there are multiple first slices, and determine the selected first slice as the target slice.

[0158] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0159] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 8 The electronic device 1200 shown includes: at least one processor 1201, a memory 1202, at least one network interface 1204 and other user interfaces 1203. The various components in the electronic device 1200 are coupled together via a bus system 1205. It is understood that the bus system 1205 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 1205 is not described in detail. Figure 8 Various buses are labeled as bus system 1205.

[0160] The user interface 1203 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0161] It can be understood that the memory 1202 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 1202 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0162] In some embodiments, the memory 1202 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 12021 and an application program 12022 .

[0163] Among them, the operating system 12021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 12022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 12022.

[0164] In the embodiment of the present application, by calling the program or instructions stored in the memory 1202, specifically, the program or instructions stored in the application 12022, the processor 1201 is used to execute the method steps provided by each method embodiment.

[0165] The method disclosed in the above embodiment of the present application can be applied to the processor 1201, or implemented by the processor 1201. The processor 1201 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1201. The above processor 1201 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1202, and the processor 1201 reads the information in the memory 1202 and completes the steps of the above method in combination with its hardware.

[0166] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0167] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0168] The electronic device provided in this embodiment may be Figure 8 The electronic device shown in FIG. 1 may perform the following steps: Figure 2-6 All steps of the method of memory management of the database system in Figure 2-6 For details, please refer to Figure 2-6 For the sake of brevity, the relevant description is not repeated here.

[0169] The embodiment of the present application also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0170] When one or more programs in the storage medium can be executed by one or more processors, the above-mentioned method of database system memory management is implemented.

[0171] The processor is used to execute the database system memory management program stored in the memory to implement the steps of the database system memory management method.

[0172] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0173] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0174] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for memory management of a database system, characterized in that: The method comprises: Regularly obtain the total memory occupied by the table cache of the leveldb instance corresponding to all shards; Determine whether the total memory value exceeds a preset table cache occupied memory total threshold; When it is determined that the total memory value exceeds the preset total memory threshold occupied by the table cache, memory data is cleaned up based on the table cache of the leveldb instance corresponding to each of the shards until the total memory value does not exceed the preset total memory threshold occupied by the table cache.

2. The method according to claim 1, characterized in that The total memory value occupied by the table cache of the leveldb instance corresponding to all shards is obtained periodically, including: Periodically obtain the data file handle of each shard cached in the table cache of the leveldb instance of each shard; Querying in memory the memory occupied value of the metadata of the data files of the shards corresponding to each of the data file handles; The total memory value is determined based on each of the occupied memory values.

3. The method according to claim 1, characterized in that The memory data cleaning based on the table cache of the leveldb instance corresponding to each of the shards includes: Obtain the data weight value corresponding to each of the shards, and the number of accesses to the table cache of the leveldb instance corresponding to each of the shards within a preset time period; Determine a target shard from the plurality of shards based on the data weight values ​​and the number of accesses corresponding to each shard; Clean up data of the data file handle of the target shard cached in the table cache of the leveldb instance corresponding to the target shard; The metadata of the data file of the target shard corresponding to the data file handle is cleaned from the memory.

4. The method according to claim 3, characterized in that The determining a target shard from the plurality of shards based on the data weight values ​​and the number of accesses corresponding to each shard comprises: For each of the shards, determining a score value of the shard according to the data weight value and the number of access times; The target shard is determined from the plurality of shards based on the score values ​​corresponding to the shards.

5. The method according to claim 3, characterized in that: The determining the target shard from the plurality of shards based on the score values ​​corresponding to the shards includes: Selecting a first score value having the smallest score value and exceeding a first preset score value from the plurality of score values; Determining whether there are multiple first score values; If it is determined that the first score value is not multiple, determining the slice corresponding to the first score value as the target slice; When it is determined that there are multiple first score values, selecting a second score value having the smallest data weight value corresponding to the slice of the first score value from the multiple first score values; determining whether the second score value is multiple; If it is determined that the second score value is not multiple, determine the slice corresponding to the second score value as the target slice; When it is determined that there are multiple second score values, one second score value is randomly selected from the multiple second score values, and the shard corresponding to the selected second score value is determined as the target shard.

6. The method according to claim 4, characterized in that The method further comprises: The score value corresponding to the target slice is reset to a second preset score value.

7. The method according to claim 3, characterized in that The determining a target shard from the plurality of shards based on the data weight values ​​and the number of accesses corresponding to each shard comprises: Selecting the first shard with the smallest data weight value and the smallest number of access times from each of the shards; Determine whether there are multiple first fragments; If it is determined that there is not more than one first shard, determining the first shard as the target shard; When it is determined that there are multiple first shards, one first shard is randomly selected from the multiple first shards, and the selected first shard is determined as the target shard.

8. A device for memory management of a database system, characterized in that: The device comprises: The acquisition module is used to periodically obtain the total memory occupied by the table cache of the leveldb instance corresponding to all shards; A determination module, used to determine whether the total memory value exceeds a preset table cache occupied total memory threshold; A data cleaning module is used to clean up the table cache of the leveldb instance corresponding to the shard when it is determined that the total memory value exceeds the total memory occupied by the preset table cache, until the total memory value does not exceed the total memory occupied by the preset table cache.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a database system memory management program stored in the memory to implement the database system memory management method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the database system memory management method according to any one of claims 1 to 7.

Images