Memory pool management method and device, storage medium, and electronic device

Abstract

The embodiment of the present application provides a method and device for managing a memory pool, a storage medium, and an electronic device, wherein the method includes: obtaining the operation requirements of a distributed storage system, and dividing the storage resources associated with the distributed storage system into large pages according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; recording historical data of using multiple target memory pools within a first preset period; constructing an intelligent prediction model through historical data and a preset initial prediction model, and using the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within a second preset period; and dynamically managing multiple target memory pools based on the adjustment requirements. The problem of high overhead and low efficiency of memory allocation, and inability to achieve dynamic management and optimization of memory resources, is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer storage technology, and more specifically, to a memory pool management method and device, a storage medium, and an electronic device. Background Art

[0002] In distributed storage systems, memory management is a crucial link. With the continuous growth of data volume and the increasing complexity of storage requirements, traditional memory management methods have been unable to meet the requirements of efficient and stable operation of distributed storage systems. Specifically, the following defects exist in related technologies: (1) Memory fragmentation: In distributed storage systems, frequent memory application and release operations, as well as the inconsistency of memory allocation size, easily lead to memory fragmentation. Fragmented memory not only reduces memory utilization, but also increases the complexity of memory management. (2) Memory leakage and memory trampling problems: Since memory applications are distributed at different stages of the I / O process, if they are not managed properly, memory leakage and memory trampling problems are very likely to occur. These problems not only affect the stability and reliability of distributed storage systems, but may also cause data loss or distributed storage system crashes. (3) Fixed-size memory object limitations: Traditional memory pool technology is usually based on fixed-size memory objects and cannot meet the dynamic requirements of distributed storage systems for memory of various sizes. This limits the flexibility and scalability of distributed storage systems. (4) Lack of intelligent prediction mechanism: Existing memory management technologies lack intelligent prediction mechanisms and cannot dynamically adjust memory allocation strategies based on the actual needs and historical data of distributed storage systems. This leads to a waste of memory resources and performance bottlenecks.

[0003] In view of the problems in related technologies such as high memory allocation overhead, low efficiency, and inability to dynamically manage and optimize memory resources, no effective solution has been proposed so far. Summary of the invention

[0004] The embodiments of the present application provide a memory pool management method and device, a storage medium, and an electronic device to at least solve the problems of high memory allocation overhead, low efficiency, and inability to achieve dynamic management and optimization of memory resources in existing methods.

[0005] According to an embodiment of the present application, a memory pool management method is provided, comprising: obtaining operation requirements of a distributed storage system, and performing large page memory partitioning on storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; recording historical data of using the multiple target memory pools within a first preset period; constructing an intelligent prediction model through the historical data and a preset initial prediction model, and using the intelligent prediction model and the current status data of the distributed storage system to predict adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current status data is used to indicate the actual usage of different memory objects in the multiple target memory pools; and dynamically managing the multiple target memory pools based on the adjustment requirements to obtain new large page memory partitioning data.

[0006] In an exemplary embodiment, the operation requirements of a distributed storage system are obtained, and storage resources associated with the distributed storage system are divided into large page memory according to the operation requirements to obtain multiple target memory pools, including: obtaining the memory level of large page memory configured for different memory objects, wherein the memory level is used to indicate the memory size range corresponding to the memory object; allocating and dividing the operation requirements according to the memory level to determine the required number of each memory object; determining the target division data of large page memory from the storage resources based on the required number and the different memory objects; and constructing the multiple target memory pools through the target division data.

[0007] In an exemplary embodiment, an intelligent prediction model is constructed using the historical data and a preset initial prediction model, including: analyzing the historical data, and determining, based on the analysis results, a usage strategy for a target memory pool corresponding to different sub-data sets under the historical data, wherein the historical data includes: multiple sub-data sets; matching the usage strategy with relevant target sub-data sets to generate multiple sets of training data; and performing machine learning training on the preset initial prediction model using the multiple sets of training data to obtain an intelligent prediction model, wherein the target sub-data set is used as a model input and the usage strategy is used as a model output.

[0008] In an exemplary embodiment, the multiple target memory pools are dynamically managed based on the adjustment demand, including: determining the type of memory object to be adjusted corresponding to the adjustment demand and the predicted target usage strategy corresponding to the type of memory object to be adjusted; calculating the current usage rate of the dynamic memory object corresponding to the type of memory object to be adjusted; when the current usage rate is less than a preset threshold and the target usage strategy is to reduce dynamic memory objects, determining an alternative memory pool for the multiple target memory pools in which the dynamic memory objects exist; determining the amount of reduction in the number of memory objects in the alternative memory pool according to a preset quantity reduction rule, and releasing a storage resource data block corresponding to the reduction to obtain first data; when the current usage rate is greater than a preset threshold and the target usage strategy is to increase dynamic memory objects, determining the remaining resource amount of the storage resource and obtaining the increased amount of the dynamic memory object; dividing the storage resource data blocks of corresponding sizes from the remaining resource amount based on the increased amount to obtain second data; determining the dynamically managed resource change amount according to the first data or the second data, and dynamically managing the multiple target memory pools through the resource change amount.

[0009] In an exemplary embodiment, before constructing an intelligent prediction model through the historical data and the preset initial prediction model, and using the intelligent prediction model and the current status data of the distributed storage system to predict the adjustment needs within the second preset period, the above method also includes: obtaining a preset monitoring strategy in the distributed storage system; determining monitoring items for the storage resources according to the monitoring strategy, wherein the monitoring items include at least: the number of existing memory pools in the storage resources, the type of memory objects corresponding to each memory pool, the number of currently used memory objects in each memory pool, and the number of free memory objects in each memory pool; determining the current status data of the distributed storage system based on the real-time monitoring data of the monitoring items.

[0010] In an exemplary embodiment, before the storage resources associated with the distributed storage system are divided into large page memory according to the operating requirements and multiple target memory pools are obtained, the above method also includes: configuring management strategies for the multiple target memory pools, wherein the management strategies include at least one of the following: dividing the memory pool into multiple memory levels according to the type of memory objects managed, and maintaining a corresponding memory pool for each size level of memory objects; determining that each memory pool contains a target number of allocated memory objects and a target number of unallocated memory objects; determining that when the distributed storage system requests to allocate memory, the memory pool selects a corresponding memory pool for allocation according to the size of the request; and determining that when a memory object is no longer needed, instructing the distributed storage system to release the memory object back to the corresponding memory pool.

[0011] In an exemplary embodiment, after dynamically managing the multiple target memory pools based on the adjustment requirements, the method further includes: obtaining the management results of the dynamic management; determining the actual change quantity of memory objects in the multiple target memory pools and the adjustment information of the memory object types of the multiple target memory pools based on the management results; sending the actual change quantity and the adjustment information to the management object corresponding to the distributed storage system, prompting the management object of the actual update result of the large page memory partition data in the second preset period.

[0012] In an exemplary embodiment, after prompting the management object with the actual update result of the large page memory partition data in the second preset period, the method further includes: receiving the display requirement of the management object for the actual update result; and adjusting the display distribution of the large page memory partition data on the visual display interface connected to the distributed storage system according to the display requirement.

[0013] In an exemplary embodiment, before building an intelligent prediction model using the historical data and a preset initial prediction model, the method further includes: identifying characteristics of the historical data and the prediction requirements of the distributed storage system; and selecting a preset initial prediction model from a model database based on the characteristics and the prediction requirements, wherein the model database includes multiple prediction models of different categories.

[0014] In an exemplary embodiment, after constructing the intelligent prediction model using the historical data and the preset initial prediction model, the method further includes: obtaining a preset verification set corresponding to the distributed storage system; performing a performance evaluation on the intelligent prediction model using the preset verification set to obtain a target evaluation score; when the target evaluation score is greater than or equal to the preset score, identifying the intelligent prediction model as a high-performance model; and when the target evaluation score is less than the preset score, identifying the intelligent prediction model as a low-performance model.

[0015] In an exemplary embodiment, the above method also includes: in a case where it is determined that the intelligent prediction model is a low-performance model, sending an update message to the distributed storage system, wherein the update message is used to instruct reselection of a preset initial prediction model from a model database; receiving a selection instruction of the distributed storage system; determining a target preset initial prediction model according to the selection instruction, and constructing a new intelligent prediction model based on the target preset initial prediction model and the historical data.

[0016] According to another embodiment of the present application, a memory pool management device is also provided, including: a partitioning module, used to obtain the operation requirements of a distributed storage system, and perform large page memory partitioning on the storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; a recording module, used to record historical data of using the multiple target memory pools within a first preset period; an adjustment module, used to build an intelligent prediction model through the historical data and a preset initial prediction model, and use the intelligent prediction model and the current status data of the distributed storage system to predict the adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current status data is used to indicate the actual usage of different memory objects in the multiple target memory pools; a management module, used to dynamically manage the multiple target memory pools based on the adjustment requirements.

[0017] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0018] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0019] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0020] Through the present application, the operation requirements of the distributed storage system are obtained, and the storage resources associated with the distributed storage system are divided into large page memory according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; the historical data of using the multiple target memory pools in a first preset period is recorded; an intelligent prediction model is constructed through the historical data and the preset initial prediction model, and the intelligent prediction model and the current state data of the distributed storage system are used to predict the adjustment requirements in a second preset period, wherein the adjustment requirements are used to determine the target memory object type and the target memory object that need to be adjusted. The type includes the number of updates of the memory object; the current status data is used to indicate the actual usage of different memory objects in multiple target memory pools; multiple target memory pools are dynamically managed based on adjustment requirements. The above technical solution is adopted to combine large page memory with multiple types of memory pools, so that the memory pool can more flexibly meet the needs of the distributed storage system for memory of different sizes, while reducing memory fragmentation and improving memory utilization. At the same time, the historical data corresponding to the memory pool is used to train and generate an intelligent prediction model, and the intelligent prediction model is used to predict the future needs of the distributed storage system for memory of different sizes and types. The adjustment of the memory pool is initiated according to future needs to realize dynamic management of multiple memory pools. Memory resources can be used more reasonably and waste of resources is avoided. The problem of high memory allocation overhead, low efficiency and inability to realize dynamic management and optimization of memory resources in the related technology is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 It is a hardware structure block diagram of a computer terminal according to a memory pool management method of an embodiment of the present application;

[0023] Figure 2 is a flow chart of a method for managing a memory pool according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a memory pool structure according to an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of the structure of a memory pool management module according to an embodiment of the present application;

[0026] Figure 5 is a structural block diagram of a memory pool management device according to an embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The memory pool management method embodiment provided in the embodiment of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of a memory pool management method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the task configuration method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a memory pool management method is provided, including but not limited to being applied in a computer terminal. Figure 2 is a flow chart of a method for managing a memory pool according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps S202-S208:

[0034] Step S202: obtaining the operation requirements of the distributed storage system, and performing large page memory division on the storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object;

[0035] Optionally, the memory pool adopts a hierarchical management mechanism, and memory objects of different sizes are allocated in different memory pools to improve memory utilization and allocation efficiency.

[0036] Step S204: Recording historical data of using the plurality of target memory pools within a first preset period;

[0037] Optionally, the distributed storage system can be preset to regularly collect memory usage, including data such as the frequency of application and release of each size of memory objects, usage duration, etc. The collected data is stored in a database for use by the intelligent prediction algorithm.

[0038] Step S206: constructing an intelligent prediction model through the historical data and the preset initial prediction model, and using the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within the second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current state data is used to indicate the actual usage of different memory objects in the multiple target memory pools;

[0039] Step S208: Dynamically manage the multiple target memory pools based on the adjustment requirements.

[0040] Optionally, in actual applications, memory objects in different target memory pools can be dynamically allocated or reclaimed according to adjustment requirements. For example, when the number of a certain type of memory objects is increased, the distributed storage system allocates new memory object blocks from the huge page memory; when the number of a certain type of memory objects is reduced, the distributed storage system releases the idle memory object blocks back to the huge page memory.

[0041] Through the above technical solution, the operation requirements of the distributed storage system are obtained, and the storage resources associated with the distributed storage system are divided into large page memory according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; the historical data of using the multiple target memory pools in a first preset period is recorded; an intelligent prediction model is constructed through the historical data and the preset initial prediction model, and the intelligent prediction model and the current state data of the distributed storage system are used to predict the adjustment requirements in the second preset period, wherein the adjustment requirements are used to determine the target memory object type and the target memory that need to be adjusted. The object type includes the number of updates of the memory object; the current status data is used to indicate the actual usage of different memory objects in multiple target memory pools; multiple target memory pools are dynamically managed based on adjustment requirements. The above technical solution is adopted to combine large page memory with multiple types of memory pools, so that the memory pool can more flexibly meet the needs of the distributed storage system for memory of different sizes, while reducing memory fragmentation and improving memory utilization. At the same time, the historical data corresponding to the memory pool is used to train and generate an intelligent prediction model, and the intelligent prediction model is used to predict the future needs of the distributed storage system for memory of different sizes and types. The adjustment of the memory pool is initiated according to future needs to realize dynamic management of multiple memory pools. Memory resources can be used more reasonably and waste of resources is avoided. The problem of high memory allocation overhead, low efficiency and inability to realize dynamic management and optimization of memory resources in the related technology is solved.

[0042] It should be noted that, through the use of large page memory and the application of intelligent prediction algorithms, this application can reduce memory fragmentation, reduce the overhead of memory application and release, and reduce the risk of memory leaks and memory stomping. These improvements not only improve the memory utilization and memory management efficiency of the distributed storage system, but also improve the overall performance and reliability of the distributed storage system. At the same time, the use of large page memory also reduces the number of page table entries, reduces the burden of kernel page table maintenance, and further improves the performance and reliability of the distributed storage system.

[0043] Optionally, the above step S202 also includes: obtaining the memory level of large page memory configured for different memory objects, wherein the memory level is used to indicate the memory size range corresponding to the memory object; allocating and dividing the operating requirements according to the memory level, and determining the required quantity of each memory object; determining the target division data of large page memory from the storage resources based on the required quantity and the different memory objects; and constructing the multiple target memory pools through the target division data.

[0044] For example, you can configure a large page memory pool. Specifically, configure a large page memory pool that contains multiple types of memory objects to meet the distributed storage system's requirements for different memory sizes. The size of the memory object ranges from 1KB to 2048KB, or larger, such as: you can set it to multiple fixed size levels, such as 1KB, 2KB, 4KB, 8KB, ..., 1024KB, 2048KB, etc. Each size level of memory objects is allocated based on large page memory to reduce memory fragmentation and improve memory utilization.

[0045] In summary, through the above implementation, the design based on the large page memory pool makes the allocation and release of memory objects more efficient, reduces the overhead of memory application and release, and at the same time reduces the memory fragmentation problem, so that the utilization rate of memory resources is effectively improved.

[0046] In an exemplary embodiment, an intelligent prediction model is constructed using the historical data and a preset initial prediction model, including: analyzing the historical data, and determining, based on the analysis results, a usage strategy for a target memory pool corresponding to different sub-data sets under the historical data, wherein the historical data includes: multiple sub-data sets; matching the usage strategy with relevant target sub-data sets to generate multiple sets of training data; and performing machine learning training on the preset initial prediction model using the multiple sets of training data to obtain an intelligent prediction model, wherein the target sub-data set is used as a model input and the usage strategy is used as a model output.

[0047] It is understandable that the above intelligent prediction model can be regarded as a practical embodiment of an intelligent prediction algorithm, which predicts the system's future demand for memory of different sizes and types by analyzing the historical memory usage data of the distributed storage system, and dynamically adjusts the number of different types of memory objects in the large page memory pool based on the prediction results. That is, by monitoring the memory usage of the distributed storage system, the demand for memory objects of different sizes in the future is predicted. The algorithm uses historical data and machine learning models to dynamically adjust the number of memory objects of each type in the memory pool to make rational use of memory resources.

[0048] In addition, the prediction algorithm corresponding to the intelligent prediction model can be but is not limited to machine learning or statistical methods, and specific implementation methods include but are not limited to linear regression, decision tree, random forest, neural network, etc.

[0049] In summary, through the above implementation, by analyzing the historical memory usage data of the distributed storage system, the future demand of the distributed storage system for memory of different sizes and types can be predicted. Based on these prediction results, the memory pool can dynamically adjust the number of memory objects of different types, thereby allocating or releasing memory resources for the distributed storage system in advance. This prediction and dynamic adjustment mechanism can significantly improve the efficiency and accuracy of memory management and reduce the risk of memory leaks and memory stagnation.

[0050] In an exemplary embodiment, the multiple target memory pools are dynamically managed based on the adjustment demand, including: determining the type of memory object to be adjusted corresponding to the adjustment demand and the predicted target usage strategy corresponding to the type of memory object to be adjusted; calculating the current usage rate of the dynamic memory object corresponding to the type of memory object to be adjusted; when the current usage rate is less than a preset threshold and the target usage strategy is to reduce dynamic memory objects, determining an alternative memory pool for the multiple target memory pools in which the dynamic memory objects exist; determining the amount of reduction in the number of memory objects in the alternative memory pool according to a preset quantity reduction rule, and releasing a storage resource data block corresponding to the reduction to obtain first data; when the current usage rate is greater than a preset threshold and the target usage strategy is to increase dynamic memory objects, determining the remaining resource amount of the storage resource and obtaining the increased amount of the dynamic memory object; dividing the storage resource data blocks of corresponding sizes from the remaining resource amount based on the increased amount to obtain second data; determining the dynamically managed resource change amount according to the first data or the second data, and dynamically managing the multiple target memory pools through the resource change amount.

[0051] Optionally, when the usage rate of a certain type of memory objects is high and the predicted demand increases, the system dynamically increases the number of memory objects of this type; when the usage rate of a certain type of memory objects is low and the predicted demand decreases, the system dynamically reduces the number of memory objects of this type.

[0052] For example, if it is predicted that the demand for a certain type of memory will increase in the future, the distributed storage system will allocate more memory objects of that type in advance; if it is predicted that the demand for a certain type of memory will decrease, the number of memory objects of that type will be reduced.

[0053] For example, when the usage rate of a certain type of memory objects exceeds a threshold and the predicted demand increases, the distributed storage system will increase the number of memory objects of this type; when the usage rate of a certain type of memory objects is lower than the threshold and the predicted demand decreases, the distributed storage system will reduce the number of memory objects of this type.

[0054] For example, when the number of a certain type of memory objects increases, the distributed storage system allocates new memory object blocks from the huge page memory; when the number of a certain type of memory objects decreases, the distributed storage system releases the free memory object blocks back to the huge page memory.

[0055] In summary, through the above implementation, the memory pool can dynamically adjust its capacity according to the results of the intelligent prediction algorithm. When it is predicted that the demand for a certain type of memory will increase, the memory pool will allocate more memory objects of this type in advance; conversely, when it is predicted that the demand will decrease, some memory objects will be released. This dynamic adjustment strategy enables memory resources to be used more reasonably and avoids waste of resources.

[0056] In an exemplary embodiment, before constructing an intelligent prediction model through the historical data and the preset initial prediction model, and using the intelligent prediction model and the current status data of the distributed storage system to predict the adjustment needs within the second preset period, the above method also includes: obtaining a preset monitoring strategy in the distributed storage system; determining monitoring items for the storage resources according to the monitoring strategy, wherein the monitoring items include at least: the number of existing memory pools in the storage resources, the type of memory objects corresponding to each memory pool, the number of currently used memory objects in each memory pool, and the number of free memory objects in each memory pool; determining the current status data of the distributed storage system based on the real-time monitoring data of the monitoring items.

[0057] Optionally, in the memory pool, the number of each type of memory objects can be dynamically adjusted based on the results of the intelligent prediction algorithm. During the adjustment process, the distributed storage system will first check the usage of each type of memory objects in the current memory pool. If the usage rate of a certain type of memory object is high and the predicted demand increases, the distributed storage system will increase the number of memory objects of this type. If the usage rate of a certain type of memory object is low and the predicted demand decreases, the distributed storage system will reduce the number of memory objects of this type.

[0058] In summary, through the above implementation, the historical memory usage data of the distributed storage system is obtained, which includes but is not limited to the size, frequency, time and other information of memory application for use by the intelligent prediction algorithm, thereby improving the compatibility of the prediction process with the actual scenario.

[0059] In an exemplary embodiment, before the storage resources associated with the distributed storage system are divided into large page memory according to the operating requirements and multiple target memory pools are obtained, the above method also includes: configuring management strategies for the multiple target memory pools, wherein the management strategies include at least one of the following: dividing the memory pool into multiple memory levels according to the type of memory objects managed, and maintaining a corresponding memory pool for each size level of memory objects; determining that each memory pool contains a target number of allocated memory objects and a target number of unallocated memory objects; determining that when the distributed storage system requests to allocate memory, the memory pool selects a corresponding memory pool for allocation according to the size of the request; and determining that when a memory object is no longer needed, instructing the distributed storage system to release the memory object back to the corresponding memory pool.

[0060] Optionally, when the distributed storage system needs memory, the memory pool allocates the most appropriate memory object according to the requested memory size, that is, the memory object type allocated from the memory pool is closest to the required memory size. When the memory is released, the memory object is returned to the corresponding memory pool for subsequent use. The management of the memory pool adopts a hierarchical management strategy, such as Figure 3 As shown, Figure 3 It is a schematic diagram of a memory pool structure according to an embodiment of the present application.

[0061] In summary, through the above implementation, large page memory is used as the basis of the memory pool, and a memory pool containing multiple types of memory objects is designed, and the sizes of these memory objects range widely, ranging from 1KB to 2048KB. This design enables the memory pool to more flexibly meet the needs of the distributed storage system for different sizes of memory, while reducing memory fragmentation and improving memory utilization.

[0062] In an exemplary embodiment, after dynamically managing the multiple target memory pools based on the adjustment requirements, the method further includes: obtaining the management results of the dynamic management; determining the actual change quantity of memory objects in the multiple target memory pools and the adjustment information of the memory object types of the multiple target memory pools based on the management results; sending the actual change quantity and the adjustment information to the management object corresponding to the distributed storage system, prompting the management object of the actual update result of the large page memory partition data in the second preset period.

[0063] In summary, through the above implementation, the number of different types of memory objects in the memory pool is adjusted based on the prediction results, so as to allocate or release memory resources for the distributed storage system in advance. At the same time, after the adjustment, the adjustment results are collected and reported to the corresponding management object of the distributed storage system to ensure that the dynamic adjustment meets the needs of the management object and reduce the risk of memory leakage and memory usage.

[0064] In an exemplary embodiment, after prompting the management object with the actual update result of the large page memory partition data in the second preset period, the method further includes: receiving the display requirement of the management object for the actual update result; and adjusting the display distribution of the large page memory partition data on the visual display interface connected to the distributed storage system according to the display requirement.

[0065] It can be understood that the adjustment status of the large page memory can be updated in real time through the above display, so that the target object can accurately know the memory changes, thereby achieving fast and reasonable allocation of memory size.

[0066] In an exemplary embodiment, before building an intelligent prediction model using the historical data and a preset initial prediction model, the method further includes: identifying characteristics of the historical data and the prediction requirements of the distributed storage system; and selecting a preset initial prediction model from a model database based on the characteristics and the prediction requirements, wherein the model database includes multiple prediction models of different categories.

[0067] It should be noted that, according to different prediction needs, the initial type of the prediction model can be flexibly selected so that the model can better identify historical data containing a type of features, ensuring the accuracy and effectiveness of the trained model in data recognition. Among them, the model can adopt model types such as time series prediction and regression analysis to improve prediction accuracy.

[0068] In an exemplary embodiment, after constructing the intelligent prediction model using the historical data and the preset initial prediction model, the method further includes: obtaining a preset verification set corresponding to the distributed storage system; performing a performance evaluation on the intelligent prediction model using the preset verification set to obtain a target evaluation score; when the target evaluation score is greater than or equal to the preset score, identifying the intelligent prediction model as a high-performance model; and when the target evaluation score is less than the preset score, identifying the intelligent prediction model as a low-performance model.

[0069] That is to say, after completing the model training, in order to ensure the accuracy of the trained model in subsequent predictions, each trained model can be scored through verification, so as to identify poorly trained models and avoid wasting training resources.

[0070] In an exemplary embodiment, the above method also includes: in a case where it is determined that the intelligent prediction model is a low-performance model, sending an update message to the distributed storage system, wherein the update message is used to instruct reselection of a preset initial prediction model from a model database; receiving a selection instruction of the distributed storage system; determining a target preset initial prediction model according to the selection instruction, and constructing a new intelligent prediction model based on the target preset initial prediction model and the historical data.

[0071] Obviously, the embodiments described above are only some embodiments of the present application, not all embodiments. In order to better understand the above method, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application, specifically:

[0072] In an exemplary embodiment, an optional embodiment of the present application proposes an intelligent dynamic memory pool management method based on large page memory for a distributed storage system. The method constructs a memory pool using large page memory and introduces an intelligent prediction algorithm to achieve dynamic management and optimization of memory resources of a distributed storage system. Specifically, the technical solution of the present application includes several aspects: (1) Memory pool based on large page memory: By constructing a memory pool using large page memory, the problem of memory fragmentation is reduced and the efficiency of memory use is improved. The memory pool contains multiple memory object types, and the size of the memory object can range from 1kb to 2048kb to meet the needs of the distributed storage system for memory of different sizes. (2) Intelligent prediction algorithm: The algorithm predicts the system's needs for memory of different sizes in the future based on the historical memory usage data of the distributed storage system, including the frequency of memory application and release, the size of memory applied and released, and other information. Based on the prediction results, the number of different memory types in the memory pool is dynamically increased or decreased to reasonably utilize memory resources. (3) Dynamic memory management: When applying for memory, a memory object of the corresponding size is allocated from the memory pool according to the size of the required memory. If there are not enough memory objects in the memory pool, the number of memory objects in the memory pool will be dynamically adjusted according to the demand predicted by the intelligent prediction algorithm. When the memory is released, the used memory objects will be recycled to the memory pool for subsequent memory requests.

[0073] Optionally, the present application designs a memory pool based on large page memory to cope with frequent memory application and release operations in distributed storage systems. The present application is based on the fact that large pages can cover a larger memory range compared to standard memory pages (such as 4KB), so the number of page table entries in the system can be significantly reduced. For example, comparing a 2MB large page with a 4KB small page, the memory space that a large page can cover is 512 times that of a small page, thereby significantly reducing the number of page table entries. Due to the reduction in the number of page table entries, the system's memory management overhead is also reduced accordingly, thereby improving the efficiency of memory access. By reducing the number of page table entries and reducing memory management overhead, large page technology can significantly improve the overall performance of the system and enhance the IO processing capabilities of distributed storage systems. In addition, the present application designs that the memory object types in the memory pool include multiple types to meet the needs of distributed storage systems for different sizes of memory. Specifically, the size of the memory object can range from 1kb to 2048kb, or even larger, to meet the memory requirements in different scenarios. The memory objects in the memory pool are classified and managed according to size to more efficiently allocate and recycle memory. The memory pool pre-allocates a certain number of memory blocks of equal size as backup. When new memory needs arise, they can be directly allocated from the memory pool without the need to allocate memory through system calls every time. This method significantly reduces the overhead of memory allocation and improves the efficiency of memory allocation. Compared with traditional memory allocation methods, such as directly using new, malloc, etc., due to the uncertain size of the memory blocks applied for, frequent use can easily cause a large amount of memory fragmentation, thereby reducing system performance. The memory blocks in the memory pool are pre-allocated and have a fixed size, which can effectively reduce the generation of memory fragmentation. In distributed storage systems, data read and write operations are very frequent, and the requirements for memory performance are very high. Using a memory pool can speed up the reading and writing of data and improve the overall performance of the system.

[0074] Optionally, in order to achieve dynamic management and optimization of memory resources, this application introduces an intelligent prediction algorithm. The algorithm predicts the system's demand for memory of different sizes and types in the future based on the historical memory usage data of the distributed storage system, including the frequency of memory application and release, the size of memory applied and released, and other information. The implementation of the intelligent prediction algorithm can be divided into the following steps. (1) Data collection: The system collects historical data on memory usage regularly or in real time, including the frequency and size of memory application and release. These data will be used as input for the prediction model. (2) Data preprocessing: The collected raw data is cleaned, converted, and standardized to remove noise and outliers and improve data quality. (3) Model selection and training: According to the characteristics of the data and the prediction requirements, a suitable prediction model is selected, such as LSTM, RNN model based on time series analysis, or regression model in machine learning. The model is trained using historical data, and the performance of the model is evaluated through a validation set. (4) Prediction: The current system status and historical data are used as inputs to the trained model to obtain the prediction results of the demand for memory of different sizes and types in the future. (5) Result processing: Process and analyze the prediction results, such as calculating the confidence interval and dynamically adjusting the number of memory objects in the memory pool based on the prediction results.

[0075] Optionally, based on the results of the intelligent prediction algorithm, the present application implements dynamic management of the memory pool. Specifically, when applying for memory, the system first searches and allocates memory objects of corresponding sizes from the memory pool according to the size of the required memory. If there are no memory objects of sufficient size in the memory pool, the number of memory objects in the memory pool is dynamically adjusted according to the demand predicted by the intelligent prediction algorithm. When the memory is released, the system recycles the used memory objects into the memory pool for subsequent memory applications. At the same time, the system will trigger the intelligent prediction algorithm again according to the size and number of the recycled memory objects to further adjust the number of memory objects in the memory pool. That is, when it is predicted that the system's demand for memory of a certain size type increases, the memory pool manager will automatically increase the number of memory objects of this type; conversely, when it is predicted that the demand decreases, the number of memory objects of this type will be reduced. By dynamically managing the number of memory objects in the memory pool, the present application can better adapt to changes in memory demand of the distributed storage system and improve the stability and reliability of the system. At the same time, since unnecessary memory allocation and recovery operations are reduced, the probability of problems such as memory leaks and memory trampling in the system can also be reduced.

[0076] As an optional implementation, the optional embodiment of the present application also provides a memory pool management module, which is tightly integrated with the I / O process, monitors and adjusts the memory allocation strategy in real time, and ensures that memory fragmentation and memory waste are minimized while meeting performance requirements. Figure 4 As shown, Figure 4 It is a structural diagram of the memory pool management module according to the embodiment of the present application, including: a large page memory pool configuration unit 402, a large page memory pool management unit 404, an intelligent prediction unit 406, and a monitoring and dynamic adjustment unit 408; further, the configuration and management of large page memory, intelligent prediction, memory usage monitoring and dynamic adjustment are realized through the above-mentioned memory pool management module. The memory pool management module is closely integrated with the I / O process of the distributed storage system, and monitors memory application and release operations in real time to ensure efficient utilization and management of the memory pool.

[0077] Optionally, the large page memory pool configuration unit 402 is mainly used to design a large page memory pool containing multiple types of memory objects to meet the needs of the distributed storage system for different sizes of memory. The size of the memory object ranges from 1KB to 2048KB, or larger, and can be specifically set to multiple fixed size levels, such as 1KB, 2KB, 4KB, 8KB, ..., 1024KB, 2048KB, etc. Memory objects of each size level are allocated based on large page memory to reduce memory fragmentation and improve memory utilization.

[0078] Optionally, the large page memory pool management unit 404 is mainly used to allocate the most appropriate memory object according to the requested memory size when the distributed storage system needs memory, that is, the type of memory object allocated from the memory pool is closest to the required memory size. When the memory is released, the memory object is returned to the corresponding memory pool for subsequent use. The management of the memory pool adopts a hierarchical management strategy. The above strategy includes: (1) dividing the memory pool into multiple levels according to the type of memory objects managed, and maintaining an independent memory pool for each size level of memory objects. (2) Each memory pool contains a certain number of allocated and unallocated memory objects. (3) When the system requests memory allocation, the memory pool selects the corresponding memory pool for allocation according to the requested size. (4) When the memory object is no longer needed, the system releases it back to the corresponding memory pool.

[0079] Optionally, the intelligent prediction unit 406 is used to predict the future demand for memory objects of different sizes by monitoring the memory usage of the distributed storage system. The algorithm uses historical data and machine learning models to dynamically adjust the number of memory objects of each type in the memory pool. The specific steps are as follows:

[0080] Step 1: Data collection: The system regularly collects memory usage, including the frequency of application and release of each size of memory objects, usage duration, etc. The collected data is stored in a database for use by the intelligent prediction algorithm.

[0081] Step 2: Data analysis: Analyze the collected data to identify patterns and trends in memory usage. For example, certain types of memory objects have increased demand during a specific time period, or are frequently used during certain operations.

[0082] Step 3: Machine learning model: Train the machine learning model based on historical data to predict future memory requirements. The model can use methods such as time series prediction and regression analysis to improve prediction accuracy.

[0083] Step 4: Dynamic adjustment: According to the prediction results, dynamically adjust the number of each type of memory objects in the memory pool. For example, if it is predicted that the demand for a certain type of memory will increase in the future, the system will allocate more memory objects of that type in advance; if it is predicted that the demand for a certain type of memory will decrease, the number of memory objects of that type will be reduced.

[0084] Optionally, the monitoring and dynamic adjustment unit 408 is mainly used to dynamically adjust the number of each type of memory objects in the memory pool according to the results of the intelligent prediction algorithm. During the adjustment process, the system will first check the usage of each type of memory objects in the current memory pool. If the usage rate of a certain type of memory object is high and the predicted demand increases, the system will increase the number of memory objects of this type. If the usage rate of a certain type of memory object is low and the predicted demand decreases, the system will reduce the number of memory objects of this type to save memory resources.

[0085] The specific steps of the monitoring and dynamic adjustment unit 408 are as follows:

[0086] Step 1: Memory usage monitoring: The system monitors the usage of various types of memory objects in real time, including the number of currently used memory objects, the number of free memory objects, etc.

[0087] Step 2: Demand forecasting: Based on the results of the intelligent forecasting algorithm, predict the demand changes of various types of memory objects in the future.

[0088] Step 3: Adjustment strategy: Develop a memory adjustment strategy based on the prediction results and current usage. For example, when the usage of a certain type of memory object exceeds the threshold and the predicted demand increases, the system will increase the number of memory objects of this type; when the usage of a certain type of memory object is lower than the threshold and the predicted demand decreases, the system will reduce the number of memory objects of this type.

[0089] Step 4: Dynamic allocation and recycling: The system dynamically allocates or recycles memory objects according to the adjustment strategy. For example, when the number of a certain type of memory objects increases, the system allocates new memory object blocks from the huge page memory; when the number of a certain type of memory objects decreases, the system releases the idle memory object blocks back to the huge page memory.

[0090] It should be noted that the above optional embodiment combines large page memory with multi-type memory pools, so that the memory pool can more flexibly meet the needs of the distributed storage system for different sizes of memory, while reducing memory fragmentation and improving memory utilization. The application of the unitized intelligent prediction algorithm can predict the future needs of the system for different sizes and types of memory by analyzing the historical memory usage data of the distributed storage system. Based on these prediction results, the memory pool can dynamically adjust the number of different types of memory objects, thereby allocating or releasing memory resources for the system in advance. This prediction and dynamic adjustment mechanism can significantly improve the efficiency and accuracy of memory management and reduce the risk of memory leaks and memory trampling. In addition, the use of dynamic adjustment strategies enables memory resources to be more reasonably utilized, avoiding waste of resources. The above improvements not only improve the memory utilization and memory management efficiency of the system, but also improve the overall performance and reliability of the system. At the same time, the use of large page memory also reduces the number of page table entries, reduces the burden of kernel page table maintenance, and further improves the performance of the system.

[0091] Through the above implementation, the design based on the large page memory pool makes the allocation and release of memory objects more efficient, reduces the overhead of memory application and release, and reduces the problem of memory fragmentation, so that the utilization rate of memory resources is effectively improved. The memory pool supports multiple types of memory objects, ranging in size from 1KB to 2048KB, which can meet the needs of distributed storage systems for different sizes of memory. The configurability of the intelligent prediction algorithm allows the system to adjust parameters according to actual conditions to adapt to different application scenarios. At the same time, the algorithm can dynamically adjust the number of memory objects in the memory pool according to the actual needs of the system, avoiding the waste of resources caused by the fixed size memory pool. Through the unified management of the memory pool, the memory leakage problem caused by frequent memory application and release is avoided. At the same time, the use of large page memory reduces the number of page table entries and reduces the risk of memory tampering caused by too many page table entries.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] In this embodiment, a memory pool management device is also provided to implement the above-mentioned embodiments and preferred implementation modes, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the modules described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0094] Figure 5 is a structural block diagram of a memory pool management device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0095] A partitioning module 52 is used to obtain the operation requirements of the distributed storage system, and perform large page memory partitioning on the storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object;

[0096] A recording module 54, configured to record historical data of using the plurality of target memory pools within a first preset period;

[0097] An adjustment module 56 is used to construct an intelligent prediction model through the historical data and the preset initial prediction model, and use the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current state data is used to indicate the actual usage of different memory objects in the multiple target memory pools;

[0098] The management module 58 is used to dynamically manage the multiple target memory pools based on the adjustment requirements.

[0099] The above-mentioned device obtains the operation requirements of the distributed storage system, and performs large page memory division on the storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; records the historical data of using the multiple target memory pools within a first preset period; constructs an intelligent prediction model through the historical data and the preset initial prediction model, and uses the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type and the target memory object that need to be adjusted. The type includes the number of updates of the memory object; the current status data is used to indicate the actual usage of different memory objects in multiple target memory pools; multiple target memory pools are dynamically managed based on adjustment requirements. The above technical solution is adopted to combine large page memory with multiple types of memory pools, so that the memory pool can more flexibly meet the needs of the distributed storage system for memory of different sizes, while reducing memory fragmentation and improving memory utilization. At the same time, the historical data corresponding to the memory pool is used to train and generate an intelligent prediction model, and the intelligent prediction model is used to predict the future needs of the distributed storage system for memory of different sizes and types. The adjustment of the memory pool is initiated according to future needs to realize dynamic management of multiple memory pools. Memory resources can be used more reasonably and waste of resources is avoided. The problem of high memory allocation overhead, low efficiency and inability to realize dynamic management and optimization of memory resources in the related technology is solved.

[0100] In an exemplary embodiment, the above-mentioned partitioning module is also used to obtain the memory level configured by the large page memory for different memory objects, wherein the memory level is used to indicate the memory size range corresponding to the memory object; the operating requirements are allocated and divided according to the memory level to determine the required number of each memory object; based on the required number and the different memory objects, the target partitioning data of the large page memory is determined from the storage resources; and the multiple target memory pools are constructed through the target partitioning data.

[0101] In an exemplary embodiment, the above-mentioned adjustment module is also used to analyze the historical data, and determine the usage strategy of the target memory pool corresponding to different sub-data sets under the historical data based on the analysis results, wherein the historical data includes: multiple sub-data sets; matching the usage strategy with the relevant target sub-data sets to generate multiple sets of training data; using the multiple sets of training data to perform machine learning training on the preset initial prediction model to obtain an intelligent prediction model, wherein the target sub-data set is used as the model input and the usage strategy is used as the model output.

[0102] In an exemplary embodiment, the management module is further used to determine the type of memory object to be adjusted corresponding to the adjustment demand and the predicted target usage strategy corresponding to the type of memory object to be adjusted; calculate the current usage rate of the dynamic memory object corresponding to the type of memory object to be adjusted; when the current usage rate is less than a preset threshold and the target usage strategy is to reduce dynamic memory objects, determine an alternative memory pool for the multiple target memory pools in which the dynamic memory objects exist; determine the amount of reduction in the number of memory objects in the alternative memory pool according to a preset quantity reduction rule, and release a storage resource data block corresponding to the reduction to obtain first data; when the current usage rate is greater than a preset threshold and the target usage strategy is to increase dynamic memory objects, determine the remaining resource amount of the storage resource and obtain the increase amount of the dynamic memory object; divide the storage resource data blocks of corresponding sizes from the remaining resource amount based on the increase amount to obtain second data; determine the resource change amount of the dynamic management according to the first data or the second data, and dynamically manage the multiple target memory pools through the resource change amount.

[0103] In an exemplary embodiment, the above-mentioned device also includes: a monitoring module, which is used to construct an intelligent prediction model through the historical data and a preset initial prediction model, and obtain a preset monitoring strategy in the distributed storage system before using the intelligent prediction model and the current status data of the distributed storage system to predict the adjustment needs within a second preset period; determine the monitoring items for the storage resources according to the monitoring strategy, wherein the monitoring items include at least: the number of existing memory pools in the storage resources, the type of memory objects corresponding to each memory pool, the number of memory objects currently used in each memory pool, and the number of free memory objects in each memory pool; determine the current status data of the distributed storage system based on the real-time monitoring data of the monitoring items.

[0104] In an exemplary embodiment, the above-mentioned device also includes: a configuration module, which is used to divide the storage resources associated with the distributed storage system into large page memory according to the operating requirements, and before obtaining multiple target memory pools, configure the management strategy of the multiple target memory pools, wherein the management strategy includes at least one of the following: dividing the memory pool into multiple memory levels according to the type of managed memory objects, and maintaining a corresponding memory pool for each size level of memory objects; determining that each memory pool contains a target number of allocated memory objects and a target number of unallocated memory objects; determining that when the distributed storage system requests to allocate memory, the memory pool selects a corresponding memory pool for allocation according to the size of the request; determining that when a memory object is no longer needed, instructing the distributed storage system to release the memory object back to the corresponding memory pool.

[0105] In an exemplary embodiment, the above-mentioned device also includes: a prompt module, which is used to dynamically manage the multiple target memory pools based on the adjustment requirements. After that, the above-mentioned method also includes: obtaining the management results of the dynamic management; determining the actual change quantity of memory objects in the multiple target memory pools and the adjustment information of the memory object types of the multiple target memory pools according to the management results; sending the actual change quantity and the adjustment information to the management object corresponding to the distributed storage system, prompting the management object of the actual update result of the large page memory partition data in the second preset period.

[0106] In an exemplary embodiment, the above-mentioned device also includes: a display module, which is used to prompt the management object of the actual update result of the large page memory partition data in the second preset period, and receive the display demand of the management object for the actual update result; adjust the display distribution of the large page memory partition data on the visual display interface connected to the distributed storage system according to the display demand.

[0107] In an exemplary embodiment, the above-mentioned device also includes: a selection module, which is used to identify the characteristics of the historical data and the prediction requirements of the distributed storage system before building an intelligent prediction model through the historical data and the preset initial prediction model; and select a preset initial prediction model from a model database based on the characteristics and the prediction requirements, wherein the model database includes multiple prediction models of different categories.

[0108] In an exemplary embodiment, the above-mentioned device also includes: an evaluation module, which is used to obtain a preset verification set corresponding to the distributed storage system after constructing an intelligent prediction model through the historical data and the preset initial prediction model; perform performance evaluation on the intelligent prediction model through the preset verification set to obtain a target evaluation score; when the target evaluation score is greater than or equal to the preset score, identify the intelligent prediction model as a high-performance model; when the target evaluation score is less than the preset score, identify the intelligent prediction model as a low-performance model.

[0109] In an exemplary embodiment, the above-mentioned device also includes: an update module, which is used to send an update message to the distributed storage system when it is determined that the intelligent prediction model is a low-performance model, wherein the update message is used to instruct to reselect a preset initial prediction model from the model database; receive a selection instruction of the distributed storage system; determine a target preset initial prediction model according to the selection instruction, and construct a new intelligent prediction model based on the target preset initial prediction model and the historical data.

[0110] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0111] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0112] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:

[0113] S1, obtaining the operation requirements of the distributed storage system, and dividing the storage resources associated with the distributed storage system into large page memory according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object;

[0114] S2, recording historical data of using the multiple target memory pools within a first preset period;

[0115] S3, constructing an intelligent prediction model through the historical data and the preset initial prediction model, and using the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current state data is used to indicate the actual usage of different memory objects in the multiple target memory pools;

[0116] S4: Dynamically manage the multiple target memory pools based on the adjustment requirements.

[0117] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0118] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, the electronic device includes a memory 502 and a processor 504. The memory 502 stores a computer program, and the processor 504 is configured to execute the steps in any of the above method embodiments through the computer program.

[0119] Optionally, in this embodiment, the processor 504 may be configured to perform the following steps through a computer program:

[0120] S1, obtaining the operation requirements of the distributed storage system, and dividing the storage resources associated with the distributed storage system into large page memory according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object;

[0121] S2, recording historical data of using the multiple target memory pools within a first preset period;

[0122] S3, constructing an intelligent prediction model through the historical data and the preset initial prediction model, and using the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current state data is used to indicate the actual usage of different memory objects in the multiple target memory pools;

[0123] S4: Dynamically manage the multiple target memory pools based on the adjustment requirements.

[0124] Alternatively, a person skilled in the art may understand that: Figure 6 The structure shown is for illustration only. Figure 6 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 6 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 6 Different configurations shown.

[0125] Among them, the memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the memory pool management method and device in the embodiment of the present application. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, that is, realizing the above-mentioned memory pool management method. The memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include a memory remotely located relative to the processor 504, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 502 can be specifically, but not limited to, used to store information such as distributed storage system configuration files. As an example, such as Figure 6 As shown, the memory 502 may include but is not limited to all modules in the memory pool management device. In addition, it may also include but is not limited to other module units in the memory pool management device, which will not be described in detail in this example.

[0126] Optionally, the transmission device 506 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 506 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0127] In addition, the electronic device mentioned above further includes: a display 508; and a connection bus 510, which is used to connect various module components in the electronic device mentioned above.

[0128] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0129] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0130] The embodiments of the present application also provide a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any one of the above method embodiments.

[0131] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0132] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for managing a memory pool, characterized in that: include: Obtaining operation requirements of a distributed storage system, and performing large page memory division on storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; Recording historical data of using the plurality of target memory pools within a first preset period; An intelligent prediction model is constructed by using the historical data and the preset initial prediction model, and the adjustment demand within a second preset period is predicted using the intelligent prediction model and the current state data of the distributed storage system, wherein the adjustment demand is used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current state data is used to indicate the actual usage of different memory objects in the multiple target memory pools; The multiple target memory pools are dynamically managed based on the adjustment requirements.

2. The memory pool management method according to claim 1, characterized in that: Obtaining the operation requirements of the distributed storage system, and performing large page memory division on the storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools, including: Obtaining memory levels configured by large page memory for different memory objects, wherein the memory level is used to indicate a memory size range corresponding to the memory object; Allocate and divide the operation requirements according to the memory level to determine the required number of each memory object; Determining target partitioning data of large page memory from the storage resource based on the required quantity and the different memory objects; The multiple target memory pools are constructed by using the target partition data.

3. The memory pool management method according to claim 1, characterized in that: Building an intelligent prediction model through the historical data and the preset initial prediction model includes: Analyze the historical data, and determine the use strategy of the target memory pool corresponding to different sub-data sets under the historical data according to the analysis result, wherein the historical data includes: multiple sub-data sets; Matching the usage strategy with relevant target sub-datasets to generate multiple sets of training data; The preset initial prediction model is trained by machine learning using the multiple sets of training data to obtain an intelligent prediction model, wherein the target sub-dataset is used as a model input and the usage strategy is used as a model output.

4. The memory pool management method according to claim 1, characterized in that: Dynamically managing the multiple target memory pools based on the adjustment requirements includes: Determine the type of memory object to be adjusted corresponding to the adjustment requirement and the predicted target usage strategy corresponding to the type of memory object to be adjusted; Calculate the current usage rate of the dynamic memory object corresponding to the memory object type to be adjusted; When the current usage rate is less than a preset threshold and the target usage policy is to reduce dynamic memory objects, determining an alternative memory pool in which the dynamic memory object exists among the multiple target memory pools; Determine a reduction amount of the number of memory objects in the candidate memory pool according to a preset number reduction rule, and release a storage resource data block corresponding to the reduction amount to obtain first data; When the current usage rate is greater than a preset threshold and the target usage policy is to increase the dynamic memory object, determine the remaining resource amount of the storage resource and obtain the increase amount of the dynamic memory object; Based on the increase amount, storage resource data blocks of corresponding sizes are divided from the remaining resource amount to obtain second data; The dynamically managed resource change amount is determined according to the first data or the second data, and the multiple target memory pools are dynamically managed through the resource change amount.

5. The memory pool management method according to claim 1, characterized in that: Before building an intelligent prediction model by using the historical data and the preset initial prediction model, and using the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment demand within the second preset period, the method further includes: Obtaining a monitoring strategy preset in the distributed storage system; Determine monitoring items for the storage resource according to the monitoring strategy, wherein the monitoring items include at least: the number of existing memory pools in the storage resource, the type of memory objects corresponding to each memory pool, the number of currently used memory objects in each memory pool, and the number of free memory objects in each memory pool; The current status data of the distributed storage system is determined based on the real-time monitoring data of the monitoring item.

6. The memory pool management method according to claim 1, characterized in that: Before dividing the storage resources associated with the distributed storage system into large page memory according to the operation requirements to obtain multiple target memory pools, the method further includes: Configure management policies for the multiple target memory pools, wherein the management policies include at least one of the following: The memory pool is divided into multiple memory levels according to the type of memory objects managed, and a corresponding memory pool is maintained for each size level of memory objects; Make sure each memory pool contains a target number of allocated memory objects and a target number of unallocated memory objects; Determine that when the distributed storage system requests memory allocation, the memory pool selects a corresponding memory pool for allocation according to the size of the request; When it is determined that the memory object is no longer needed, the distributed storage system is instructed to release the memory object back to the corresponding memory pool.

7. The memory pool management method according to claim 1, characterized in that: After dynamically managing the multiple target memory pools based on the adjustment requirements, the method further includes: Obtaining the management result of the dynamic management; Determine, according to the management result, the actual changed quantity of the memory objects in the multiple target memory pools and the adjustment information of the memory object types of the multiple target memory pools; The actual change quantity and the adjustment information are sent to a management object corresponding to the distributed storage system, prompting the management object with an actual update result of the large page memory partition data in the second preset period.

8. The memory pool management method according to claim 7, characterized in that: After prompting the management object with the actual update result of the large page memory partition data in the second preset period, the method further includes: receiving a display requirement of the management object for the actual update result; The display distribution of the large page memory partition data on the visualization display interface connected to the distributed storage system is adjusted according to the display requirement.

9. The memory pool management method according to claim 7, characterized in that: Before building the intelligent prediction model using the historical data and the preset initial prediction model, the method further includes: Identifying characteristics of the historical data and predicted requirements of the distributed storage system; A preset initial prediction model is selected from a model database according to the characteristics and the prediction requirements, wherein the model database includes a plurality of prediction models of different categories.

10. The memory pool management method according to claim 7, characterized in that: After constructing the intelligent prediction model through the historical data and the preset initial prediction model, the method further includes: Obtaining a preset verification set corresponding to the distributed storage system; Performing performance evaluation on the intelligent prediction model through the preset validation set to obtain a target evaluation score; When the target evaluation score is greater than or equal to a preset score, identifying the intelligent prediction model as a high-performance model; When the target evaluation score is less than a preset score, the intelligent prediction model is identified as a low-performance model.

11. The memory pool management method according to claim 10, characterized in that: The method further comprises: In the case where it is determined that the intelligent prediction model is a low-performance model, an update message is sent to the distributed storage system, wherein the update message is used to instruct to reselect a preset initial prediction model from the model database; Receiving a selection instruction of the distributed storage system; A target preset initial prediction model is determined according to the selection instruction, and a new intelligent prediction model is constructed based on the target preset initial prediction model and the historical data.

12. A memory pool management device, characterized in that: include: A partitioning module is used to obtain the operation requirements of the distributed storage system, and perform large page memory partitioning on the storage resources associated with the distributed storage system according to the operation requirements to obtain multiple target memory pools; wherein each of the multiple target memory pools corresponds to a class of memory object types, and the class of memory object types includes at least one memory object; A recording module, used for recording historical data of using the plurality of target memory pools within a first preset period; An adjustment module, configured to construct an intelligent prediction model through the historical data and a preset initial prediction model, and use the intelligent prediction model and the current state data of the distributed storage system to predict the adjustment requirements within a second preset period, wherein the adjustment requirements are used to determine the target memory object type that needs to be adjusted and the target memory object type includes the number of updates of the memory object; the current state data is used to indicate the actual usage of different memory objects in the multiple target memory pools; A management module is used to dynamically manage the multiple target memory pools based on the adjustment requirements.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 11 when executed by a processor.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 11 through the computer program.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images