Memory management method and device

By building a memory estimate model and configuring the heap memory capacity according to the frequency of full heap garbage collection, the problem of low adaptation efficiency of heap memory capacity in the existing technology is solved, efficient and accurate memory management is achieved, and resource waste and business impact is reduced. .

CN118916174APending Publication Date: 2024-11-08BEIJING VOLCANO ENGINE TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411117843.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently and accurately adapt the heap memory capacity, resulting in waste of resources or affecting business performance and stability, and requires professional knowledge and business-by-business optimization, which is inefficient.

Method used

By obtaining the log data of garbage collection for heap memory, obtaining indicators related to the memory usage of the elderly, building a memory estimate model with the full heap garbage collection frequency as the independent variable and the estimated heap memory capacity as the dependent variable, and the expected full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time full-time The heap garbage collection frequency obtains the target heap memory capacity and configures the heap memory capacity.

Benefits of technology

Efficiently and accurately adapt the heap memory capacity without professional knowledge, reducing resource waste and business impact, and improving memory management efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916174A_ABST
    Figure CN118916174A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a memory management method and device, and relates to the technical field of computer science. The method comprises the following steps: acquiring log data for executing garbage collection on a heap memory; obtaining indexes related to the memory usage amount of the old age according to the log data; according to the indexes related to the old memory usage amount and the original heap memory capacity of the heap memory, constructing a memory estimation model which takes the full-heap garbage collection frequency as an independent variable and estimates the heap memory capacity as a dependent variable; obtaining a target heap memory capacity according to the memory prediction model and an expected full heap garbage collection frequency; and configuring the capacity of the heap memory according to the target heap memory capacity. The embodiment of the invention is used for efficiently and accurately adapting the heap memory capacity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer science and technology, and in particular to a memory management method and device. Background Art

[0002] With the continuous development of big data technology and the increasing demand for applications, big data has placed higher and higher demands on memory management and optimization, especially on heap memory management. Due to the complexity of heap memory management and the diversity of big data application scenarios, how to efficiently and accurately adapt heap memory capacity has become a technical challenge. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a memory management method and device for efficiently and accurately adapting the heap memory capacity.

[0004] In order to achieve the above purpose, the embodiments of the present application provide the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a memory management method, including:

[0006] Get the log data of garbage collection performed on heap memory;

[0007] Obtaining indicators related to old generation memory usage according to the log data;

[0008] According to the indicators related to the old generation memory usage and the original heap memory capacity corresponding to the log data, a memory estimation model is constructed with the full heap garbage collection frequency as an independent variable and the estimated heap memory capacity as a dependent variable;

[0009] Obtaining a target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency;

[0010] The capacity of the heap memory is configured according to the target heap memory capacity.

[0011] As an optional implementation of the embodiment of the present application, the obtaining of an indicator related to the memory usage of the old generation according to the log data includes:

[0012] Obtaining historical promotion rate, historical application rate and historical survival capacity according to the log data;

[0013] Among them, the historical promotion rate is the rate at which the young generation memory is promoted to the old generation memory; the historical application rate is the application rate of the old generation memory; and the historical survival capacity is the size of the surviving objects after the full heap garbage collection.

[0014] As an optional implementation of the embodiment of the present application, the memory estimation model with full heap garbage collection frequency as an independent variable and estimated heap memory capacity as a dependent variable is constructed according to the indicator related to the old generation memory usage and the original heap memory capacity of the heap memory, including:

[0015] Constructing a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate according to the original heap memory capacity and the historical promotion rate;

[0016] Constructing a second conversion relationship between the full-heap garbage collection frequency and the estimated heap memory capacity according to the historical surviving capacity, the historical application rate, and the estimated promotion rate;

[0017] The memory estimation model is constructed according to the first conversion relationship and the second conversion relationship.

[0018] As an optional implementation of the embodiment of the present application, constructing a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate according to the original heap memory capacity and the historical promotion rate includes:

[0019] Acquire a first parameter according to a change in the estimated heap memory capacity relative to the original heap memory capacity;

[0020] The historical promotion rate is compensated according to the first parameter to obtain the first conversion relationship.

[0021] As an optional implementation of the embodiment of the present application, the second conversion relationship between the full heap garbage collection frequency and the estimated heap memory capacity is constructed according to the historical survival capacity, the historical application rate, and the estimated promotion rate, including:

[0022] Obtaining the memory usage of the first old generation according to the historical application rate and the full-heap garbage collection frequency;

[0023] Obtaining the memory usage of the second old generation according to the estimated promotion rate and the full-heap garbage collection frequency;

[0024] Constructing a conversion relationship between the old generation memory usage and the full heap garbage collection frequency according to the first old generation memory usage, the second old generation memory usage, and the historical surviving capacity;

[0025] The second conversion relationship is generated according to the conversion relationship between the old generation memory usage and the full heap garbage collection frequency.

[0026] As an optional implementation of the embodiment of the present application, configuring the capacity of the heap memory according to the target heap memory capacity includes:

[0027] Acquire an upper boundary and a lower boundary of the heap memory capacity according to the original heap memory capacity;

[0028] Determine whether the target heap memory capacity is greater than the lower limit and less than the upper limit;

[0029] If the target heap memory capacity is greater than the lower limit and less than the upper limit, configuring the target heap memory capacity as the capacity of the heap memory;

[0030] If the target heap memory capacity is less than or equal to the lower limit, configuring the lower limit as the capacity of the heap memory;

[0031] If the target heap memory capacity is greater than or equal to the upper limit, the upper limit is configured as the capacity of the heap memory.

[0032] As an optional implementation of the embodiment of the present application, configuring the capacity of the heap memory according to the target heap memory capacity includes:

[0033] Acquire an upper boundary and a lower boundary of the heap memory capacity according to the original heap memory capacity;

[0034] Normalizing the target heap memory capacity based on the upper boundary and the lower boundary to obtain a normalized value of the target heap memory capacity;

[0035] Obtaining a denormalized value of the target heap memory capacity according to the normalized value, the upper boundary, and the lower boundary;

[0036] The denormalized value is configured as the capacity of the heap memory.

[0037] As an optional implementation of the embodiment of the present application, the obtaining of an indicator related to the memory usage of the old generation according to the log data includes:

[0038] Parsing the log data to obtain historical garbage collection events;

[0039] Using the K-Means classification algorithm, the historical garbage collection events are clustered into two clusters based on the frequency of occurrence;

[0040] An indicator related to the memory usage of the old generation is obtained through historical garbage collection events in the cluster with a higher occurrence frequency among the two clusters.

[0041] As an optional implementation of the embodiment of the present application, the obtaining of log data of garbage collection performed on the heap memory includes:

[0042] Obtain log data of garbage collection performed on the heap memory of the Java virtual machine in the stream processing business system.

[0043] In a second aspect, an embodiment of the present application provides a memory management device, including:

[0044] An acquisition unit, used for acquiring log data of garbage collection performed on heap memory;

[0045] A parsing unit, used to obtain indicators related to the memory usage of the old generation according to the log data;

[0046] A construction unit, configured to construct a memory estimation model with full heap garbage collection frequency as an independent variable and estimated heap memory capacity as a dependent variable according to an indicator related to the old generation memory usage and an original heap memory capacity of the heap memory;

[0047] A processing unit, configured to obtain a target heap memory capacity according to the memory estimation model and an expected full heap garbage collection frequency;

[0048] A configuration unit is used to configure the capacity of the heap memory according to the target heap memory capacity.

[0049] As an optional implementation of the embodiment of the present application, the parsing unit is specifically used to obtain the historical promotion rate, the historical application rate and the historical survival capacity according to the log data;

[0050] Among them, the historical promotion rate is the rate at which the young generation memory is promoted to the old generation memory; the historical application rate is the application rate of the old generation memory; and the historical survival capacity is the size of the surviving objects after the full heap garbage collection.

[0051] As an optional implementation of an embodiment of the present application, the construction unit is specifically used to construct a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate based on the original heap memory capacity and the historical promotion rate; construct a second conversion relationship between the full heap garbage collection frequency and the estimated heap memory capacity based on the historical survival capacity, the historical application rate, and the estimated promotion rate; and construct the memory estimation model based on the first conversion relationship and the second conversion relationship.

[0052] As an optional implementation of an embodiment of the present application, the construction unit is specifically used to obtain a first parameter based on a change in the estimated heap memory capacity relative to the original heap memory capacity; and compensate the historical promotion rate based on the first parameter to obtain the first conversion relationship.

[0053] As an optional implementation manner of the embodiment of the present application, the construction unit is specifically used to obtain the first old generation memory usage based on the historical application rate and the full-heap garbage collection frequency; obtain the second old generation memory usage based on the estimated promotion rate and the full-heap garbage collection frequency; construct a conversion relationship between the old generation memory usage and the full-heap garbage collection frequency based on the first old generation memory usage, the second old generation memory usage and the historical surviving capacity; generate the second conversion relationship based on the conversion relationship between the old generation memory usage and the full-heap garbage collection frequency.

[0054] As an optional implementation of the embodiment of the present application, the configuration unit is specifically used to obtain the upper and lower boundaries of the heap memory capacity based on the original heap memory capacity; determine whether the target heap memory capacity is greater than the lower boundary and less than the upper boundary; if the target heap memory capacity is greater than the lower boundary and less than the upper boundary, configure the target heap memory capacity as the capacity of the heap memory; if the target heap memory capacity is less than or equal to the lower boundary, configure the lower boundary as the capacity of the heap memory; if the target heap memory capacity is greater than or equal to the upper boundary, configure the upper boundary as the capacity of the heap memory.

[0055] As an optional implementation of an embodiment of the present application, the configuration unit is specifically used to obtain the upper boundary and the lower boundary of the heap memory capacity based on the original heap memory capacity; normalize the target heap memory capacity based on the upper boundary and the lower boundary to obtain a normalized value of the target heap memory capacity; obtain an inverse normalized value of the target heap memory capacity based on the normalized value, the upper boundary and the lower boundary; and configure the inverse normalized value as the capacity of the heap memory.

[0056] As an optional implementation of an embodiment of the present application, the parsing unit is specifically used to parse the log data to obtain historical garbage collection events; use the K-Means classification algorithm to cluster the historical garbage collection events into two clusters based on the frequency of occurrence; and obtain indicators related to the old generation memory usage through the historical garbage collection events in the cluster with a higher occurrence frequency of the two clusters.

[0057] As an optional implementation of the embodiment of the present application, the acquisition unit is specifically used to obtain log data of garbage collection executed by the JVM in the stream processing business system.

[0058] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the electronic device to implement the memory management method described in any of the above embodiments when executing the computer program.

[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when the computer program is executed by a computing device, enables the computing device to implement any of the above-mentioned memory management methods.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to implement any of the above-mentioned memory management methods.

[0061] The memory management method provided in the embodiment of the present application first obtains the log data of garbage collection performed on the heap memory, and obtains the index related to the memory usage of the old generation according to the log data, and then constructs a memory estimation model with the full heap garbage collection frequency as the independent variable and the estimated heap memory capacity as the dependent variable according to the index related to the memory usage of the old generation and the original heap memory capacity of the heap memory, and then obtains the target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency, and configures the capacity of the heap memory according to the target heap memory capacity. Since the memory management method provided in the embodiment of the present application can construct a memory estimation model with the full heap garbage collection frequency as the variable and the estimated heap memory capacity as the dependent variable according to the index related to the memory usage of the old generation and the original heap memory capacity, after setting the expected full heap garbage collection frequency, the corresponding target heap memory capacity can be obtained according to the memory estimation model, and the capacity of the heap memory can be configured according to the target heap memory capacity, without the user having the relevant background knowledge of memory management, and without optimizing one by one for a single business, the above embodiment can efficiently and accurately adapt the heap memory capacity. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0064] Figure 1 One of the interactive flow charts of the memory management method provided in the embodiment of the present application;

[0065] Figure 2 The second interactive flow chart of the memory management method provided in the embodiment of the present application;

[0066] Figure 3A schematic diagram of the structure of a memory management device provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0069] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0070] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0071] When configuring heap memory, if the configured memory resources are too large, it will cause resource waste and increase system deployment costs; if the configured memory resources are too small, it will affect business performance and stability. At present, heap memory configuration and optimization mainly rely on personal experience and trial and error. On the one hand, this requires users to have relevant knowledge of memory management, and the professional threshold is high; on the other hand, heap memory optimization often requires repeated adjustments and tests, and can only be optimized for a single business one by one, which is inefficient and difficult to cope with large-capacity stream processing business scenarios. Therefore, how to efficiently and accurately adapt the heap memory capacity is an urgent problem to be solved.

[0072] The present application embodiment provides a memory management method, referring to Figure 1 As shown, the memory management method includes the following steps:

[0073] S11. Obtain log data of garbage collection (GC) performed on heap memory.

[0074] Garbage collection is a memory management mechanism. When the memory occupied by objects that are no longer used needs to be reclaimed for reuse, the program will perform garbage collection. The main purpose of garbage collection is to release the memory space occupied by objects that are no longer referenced to avoid memory leaks and improve memory usage efficiency.

[0075] The heap memory in the embodiment of the present application may be a memory area allocated to a specific object for dynamic data storage. For example, the heap memory may be the heap memory allocated by the Java Virtual Machine (JVM) in Java; another example, the heap memory may be the heap memory allocated by objects, arrays, etc. in C or C++; another example, the heap memory may be the heap memory allocated by the image processing program for storing image data. The embodiment of the present application does not limit the object to which the heap memory belongs and the programming language used by the object.

[0076] In some embodiments, the log data of executing garbage collection may include: the number of garbage collections, the time of garbage collection, the heap memory usage before garbage collection, the heap memory usage after garbage collection, the efficiency index of garbage collection, etc. One or more of the following.

[0077] In some embodiments, the above step S11 (obtaining log data of garbage collection executed on heap memory) includes: obtaining log data of garbage collection executed on heap memory of a Java Virtual Machine (JVM) in the stream processing business system.

[0078] That is, the log data is log data generated by garbage collection performed by the JVM in the stream processing business system. The memory management method provided in the embodiment of the present application can estimate and configure the heap memory of the JVM of the stream processing business system.

[0079] In some embodiments, the stream processing business system may be an Apache Flink business system.

[0080] S12. Obtain indicators related to the memory usage of the old generation (Old Generation) according to the log data.

[0081] Under the generational memory management mechanism, heap memory is generally divided into young generation memory and old generation memory for separate management. The young generation memory is managed using a replication algorithm and is divided into the initial (Eden) area and two survivor (Survivor) areas; when the Eden area is full, Minor GC (young generation memory garbage collection) will be triggered, and the surviving objects in the Eden area will be copied from the Eden area to the first Survivor area, and the surviving objects in the first Survivor area will be copied from the Eden area to the second Survivor area. Objects that are still alive after multiple Minor GCs and have reached a preset age will be promoted to the old generation memory. The old generation memory is managed using a compression algorithm. As the old generation memory continues to grow, the old generation memory will reach a set threshold. When the set threshold is reached, a full heap garbage collection (Full GC) will be triggered to recycle and reuse the memory space occupied by dead objects in the entire heap memory (including: young generation memory and old generation memory).

[0082] The memory usage of the old generation includes three parts: 1. The memory occupied by surviving objects after full heap garbage collection; 2. The memory occupied by objects promoted from the young generation memory to the old generation memory; 3. The memory occupied by Humongous (large objects, objects directly requested in the old generation memory); therefore, indicators related to the memory usage of the old generation may include indicators related to these three parts, such as: the memory occupied by surviving objects after full heap garbage collection, the rate at which surviving objects are discarded after full heap garbage collection, the rate at which Humongous are requested, etc.

[0083] S13. According to the indicator related to the old generation memory usage and the original heap memory capacity of the heap memory, a memory estimation model is constructed with the full heap garbage collection frequency as the independent variable and the estimated heap memory capacity as the dependent variable.

[0084] Since full-heap garbage collection needs to be performed after all business threads are paused, and all objects in the heap memory need to be marked, the duration of full-heap garbage collection is much longer than that of Minor GC, and has a greater impact on the business. When full-heap garbage collection is triggered too frequently, such as once every few seconds, it will have a great impact on the business, indicating that the business is in an unhealthy state; when the frequency of full-heap garbage collection is too low, such as only once a day, it indicates that there is a lot of room for optimization of the heap memory. Therefore, the frequency of full-heap garbage collection is a key indicator of whether the business is healthy. Constructing a memory estimation model with full-heap garbage collection frequency as the independent variable and estimated heap memory capacity as the dependent variable can accurately estimate the required capacity of heap memory.

[0085] S14. Obtain a target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency.

[0086] That is, a desired full-heap garbage collection frequency is preset, and the desired full-heap garbage collection frequency is substituted into the memory estimation model to obtain a target heap memory capacity.

[0087] In some embodiments, the expected full-heap garbage collection frequency can be set to balance the requirements of business performance and stability and the memory deployment cost. The higher the expected full-heap garbage collection frequency, the lower the memory deployment cost, but the worse the business performance and stability. Conversely, the lower the expected full-heap garbage collection frequency, the better the business performance and stability, but the higher the memory deployment cost.

[0088] S15. Configure the capacity of the heap memory according to the target heap memory capacity.

[0089] In some embodiments, configuring the capacity of the heap memory according to the target heap memory capacity includes:

[0090] The target heap memory capacity is configured as the capacity of the heap memory. For example, if the target heap memory capacity is 20 MB, the heap memory capacity is configured as 20 MB.

[0091] The memory management method provided in the embodiment of the present application first obtains the log data of garbage collection performed on the heap memory, and obtains the index related to the memory usage of the old generation according to the log data, and then constructs a memory estimation model with the full heap garbage collection frequency as the independent variable and the estimated heap memory capacity as the dependent variable according to the index related to the memory usage of the old generation and the original heap memory capacity of the heap memory, and then obtains the target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency, and configures the capacity of the heap memory according to the target heap memory capacity. Since the memory management method provided in the embodiment of the present application can construct a memory estimation model with the full heap garbage collection frequency as the variable and the estimated heap memory capacity as the dependent variable according to the index related to the memory usage of the old generation and the original heap memory capacity, after setting the expected full heap garbage collection frequency, the corresponding target heap memory capacity can be obtained according to the memory estimation model, and the capacity of the heap memory can be configured according to the target heap memory capacity, without the user having the relevant background knowledge of memory management, and without optimizing one by one for a single business, the above embodiment can efficiently and accurately adapt the heap memory capacity.

[0092] As an extension and refinement of the above embodiment, the present application embodiment provides another memory management method, referring to Figure 2 As shown, the memory management method includes the following steps:

[0093] S201. Obtain log data of garbage collection performed on heap memory.

[0094] S202: Parse the log data to obtain historical garbage collection events.

[0095] In some embodiments, the log data may be parsed using a CG parsing tool to obtain the historical garbage collection events.

[0096] S203: Using a K-Means classification algorithm, the historical garbage collection events are clustered into two clusters based on the occurrence frequency.

[0097] The K-Means classification algorithm is an unsupervised learning algorithm used to cluster a dataset into a specified number (K) of clusters based on clustering features.

[0098] In some embodiments, the historical garbage collection events are clustered into two clusters based on the occurrence frequency using a K-Means classification algorithm, including the following steps a to d:

[0099] Step a: Randomly select two historical garbage collection events as the initial cluster centers.

[0100] Step b: For each historical garbage collection event, the distances from the historical garbage collection event to two cluster centers are calculated based on the occurrence frequency, and the event is assigned to the cluster to which the cluster center closest to the event belongs.

[0101] Step c: Recalculate the means of all historical garbage collection events in the two clusters as new cluster centers.

[0102] Step d: Repeat steps b and c until the cluster center no longer changes significantly or the predetermined number of iterations is reached.

[0103] S204: Obtain a historical promotion rate, a historical application rate, and a historical survival capacity through historical garbage collection events in the cluster with a higher occurrence frequency among the two clusters.

[0104] Among them, the historical promotion rate is the rate at which young generation memory is promoted to old generation memory; the historical application rate is the application rate of old generation memory; and the historical survival capacity is the size of surviving objects after full heap garbage collection.

[0105] Businesses generally have obvious peak and trough characteristics. The above embodiment clusters the historical garbage collection events into two clusters based on the frequency of occurrence, and obtains the historical promotion rate, historical application rate and historical survival capacity through the historical garbage collection events in the cluster with higher occurrence frequency of the two clusters. Therefore, the above embodiment can make the obtained target heap memory capacity meet the performance requirements during business peak periods.

[0106] In some embodiments, the historical promotion rate may be a ratio of the total amount of young generation memory promoted to old generation memory in a specified historical time period to the length of the specified historical time period, in MB / minute.

[0107] In some embodiments, the historical application rate may be a ratio of the total amount of memory requested by Humongous (objects requested directly in the old generation memory) in a specified historical time period to the length of the specified historical time period, in MB / minute.

[0108] In some embodiments, the historical survival capacity may be an average value of the size of the surviving objects after each full heap garbage collection, in MB. For example, if 4 full heap garbage collections are performed, and the sizes of the surviving objects after the 4 full heap garbage collections are A, B, C, and D, respectively, the historical survival capacity may be (A+B+C+D) / 4.

[0109] S205: Construct a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate according to the original heap memory capacity and the historical promotion rate.

[0110] After the heap memory capacity is adjusted, the object promotion rate will change while the business object application rate remains unchanged. The changing trend is: when the heap memory capacity increases, the rate at which the young generation memory is promoted to the old generation memory will decrease, and when the heap memory capacity decreases, the rate at which the young generation memory is promoted to the old generation memory will increase. Therefore, compensation can be made based on the historical promotion rate to obtain the conversion relationship between the changed heap memory capacity (estimated heap memory capacity) and the estimated promotion rate.

[0111] In some embodiments, constructing a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate according to the original heap memory capacity and the historical promotion rate includes the following steps 2051 and 2052:

[0112] Step 2051: Obtain a first parameter according to a change in the estimated heap memory capacity relative to the original heap memory capacity.

[0113] In some embodiments, obtaining a first parameter according to a change in the estimated heap memory capacity relative to the original heap memory capacity includes obtaining the first parameter according to the following formula:

[0114]

[0115] Among them, G% is the first parameter, HeapEst is the estimated heap memory capacity, and HeapOrig is the original heap memory capacity.

[0116] Step 2052: Compensate the historical promotion rate according to the first parameter to obtain the first conversion relationship.

[0117] In some embodiments, the first conversion relationship can be expressed as:

[0118]

[0119] Among them, PR Adjust is the estimated promotion rate, PR Orig is the historical promotion rate, HeapEst is the estimated heap memory capacity, HeapOrig is the original heap memory capacity, and β is a constant.

[0120] In some embodiments, β may be an empirical parameter set by relevant technicians.

[0121] In some embodiments, the first conversion relationship may also be transformed, for example, the first conversion relationship may be transformed into:

[0122]

[0123] However, this variation is merely an equivalent replacement of the first conversion relationship, and does not make the essence of the first conversion relationship deviate from the scope of the technical solution of the above embodiment.

[0124] S206: Construct a second conversion relationship between the full-heap garbage collection frequency and the estimated heap memory capacity according to the historical surviving capacity, the historical application rate, and the estimated promotion rate.

[0125] As mentioned above, the old generation memory usage includes three parts: 1. The memory occupied by surviving objects after full heap garbage collection; 2. The memory occupied by objects promoted from young generation memory to old generation memory; 3. The memory requested by Humongous. Therefore, the memory occupied by these three parts is summed up, and then the required capacity of the heap memory is obtained by multiplying the sum by a coefficient based on the ratio of the capacity of the old generation memory to the capacity of the young generation memory. Therefore, the second conversion relationship between the full heap garbage collection frequency and the estimated heap memory capacity can be constructed based on the historical surviving capacity, the historical request rate, and the estimated promotion rate.

[0126] In some embodiments, constructing a second conversion relationship between the full heap garbage collection frequency and the estimated heap memory capacity according to the historical surviving capacity, the historical application rate, and the estimated promotion rate includes the following steps 2061 to 2064:

[0127] Step 2061: Obtain the memory usage of the first old generation according to the historical application rate and the full-heap garbage collection frequency.

[0128] The historical request rate is represented as HR, and the full-heap garbage collection frequency is represented as P FullGC , the memory usage of the first old generation is represented by H Old , then:

[0129] H Old =HR*P FullGC

[0130] Step 2062: Obtain the memory usage of the first old generation according to the estimated promotion rate and the full-heap garbage collection frequency.

[0131] Express the estimated promotion rate as PR Adjust , the full heap garbage collection frequency is denoted as P FullGC , the memory usage of the first old generation is represented by H Old , then:

[0132] O Old =PR Adjust *P FullGC

[0133] Step 2063: construct a conversion relationship between the old generation memory usage and the full heap garbage collection frequency according to the first old generation memory usage, the second old generation memory usage, and the historical survival capacity.

[0134] As mentioned above, the memory usage of the old generation includes three parts: 1. The memory occupied by surviving objects after full-heap garbage collection; 2. The memory occupied by objects promoted from the young generation memory to the old generation memory; 3. The memory requested by Humongous. Therefore, the conversion relationship between the memory usage of the old generation and the full-heap garbage collection frequency can be:

[0135] T Old =A FullGC +H Old +P Old =A FullGC +(HR+PR Adjust )*P FullGC

[0136] Among them, T Old is the memory usage of the old generation, A FullGC is the historical survival capacity, HR is the historical application rate, PR Adjust is the estimated promotion rate, P FullGC is the full heap garbage collection frequency.

[0137] Step 2064: Generate the second conversion relationship according to the conversion relationship between the old generation memory usage and the full heap garbage collection frequency.

[0138] In some embodiments, the second conversion relationship is generated based on the conversion relationship between the old generation memory usage and the full heap garbage collection frequency, including: the proportion of the old generation memory in the heap memory and the conversion relationship between the old generation memory usage and the full heap garbage collection frequency, to generate the second conversion relationship.

[0139] In some embodiments, the proportion of the old generation memory in the heap memory is 2 / 3, and the second conversion relationship may be:

[0140] HeapEst=[A FullGC +(HR+PR Adjust )*P FullGC ]*1.5

[0141] Among them, HeapEst is the estimated heap memory capacity, A FullGC is the historical survival capacity, HR is the historical application rate, PR Adjust is the estimated promotion rate, P FullGC is the full heap garbage collection frequency.

[0142] In some embodiments, the second conversion relationship may also be transformed, for example, the second conversion relationship may be transformed into:

[0143]

[0144] Among them, C FullGC The number of full heap garbage collections per day (every 24 hours).

[0145] S207: Construct the memory estimation model according to the first conversion relationship and the second conversion relationship.

[0146] Combining the first conversion relationship and the second conversion relationship, a quadratic equation for estimating the heap memory capacity can be obtained. The quadratic equation for estimating the heap memory capacity is the memory estimation model. Substituting known quantities such as the historical survival capacity, the historical application rate, and the estimated promotion rate into the quadratic equation can obtain a quadratic equation with the full-heap garbage collection frequency as the independent variable and the estimated heap memory capacity as the dependent variable.

[0147] In some embodiments, the memory estimation model is:

[0148]

[0149] Where HeapEst is the estimated heap memory capacity, HeapOrig is the original heap memory capacity, and A FullGC is the historical survival capacity, P FullGCis the full-heap garbage collection frequency, HR is the historical application rate, PR Orig is the historical promotion rate, and β is a constant.

[0150] Similarly, the memory estimation model may be transformed, for example, the memory estimation model may be transformed into:

[0151]

[0152] S208. Obtain a target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency.

[0153] As described above, by substituting the known quantities such as the historical surviving capacity, the historical application rate, and the estimated promotion rate into the quadratic equation, a quadratic equation with the full-heap garbage collection frequency as the independent variable and the estimated heap memory capacity as the dependent variable can be obtained. By substituting the expected full-heap garbage collection frequency into the quadratic equation, the quadratic equation can be solved to obtain the heap memory capacity corresponding to the expected full-heap garbage collection frequency.

[0154] S209: configuring the capacity of the heap memory according to the target heap memory capacity.

[0155] In some embodiments, configuring the capacity of the heap memory according to the target heap memory capacity includes: configuring the target heap memory capacity as the capacity of the heap memory.

[0156] In some embodiments, configuring the capacity of the heap memory according to the target heap memory capacity includes the following steps a to e:

[0157] Step a: obtaining an upper boundary and a lower boundary of the heap memory capacity according to the original heap memory capacity.

[0158] In some embodiments, the upper limit of the heap memory capacity is (1+30%)*original heap memory capacity, and the lower limit of the heap memory capacity is (1-30%)*original heap memory capacity. For example, if the original heap memory capacity is 10MB, the upper limit and lower limit of the heap memory capacity are 13MB and 7MB respectively.

[0159] Step b: Determine whether the target heap memory capacity is greater than the lower boundary and less than the upper boundary.

[0160] In the above step b, if the target heap memory capacity is greater than the lower limit and less than the upper limit, execute the following step c; if the target heap memory capacity is less than or equal to the lower limit, execute the following step d; if the target heap memory capacity is greater than or equal to the upper limit, execute the following step e:

[0161] Step c: configuring the target heap memory capacity to be the capacity of the heap memory.

[0162] Step d: configuring the lower boundary to be the capacity of the heap memory.

[0163] Step e: configuring the upper boundary to be the capacity of the heap memory.

[0164] In the above embodiment, when the target heap memory capacity is less than or equal to the lower boundary, the lower boundary is configured as the capacity of the heap memory; when the target heap memory capacity is less than or equal to the lower boundary, the lower boundary is configured as the capacity of the heap memory; when the target heap memory capacity is greater than or equal to the upper boundary, the upper boundary is configured as the capacity of the heap memory. Therefore, the configured heap memory capacity is always within the range of the upper and lower boundaries of the heap memory capacity. Therefore, the above embodiment can avoid excessive adjustment of the heap memory capacity, which in turn affects the accuracy of object promotion rate estimation.

[0165] In some embodiments, configuring the capacity of the heap memory according to the target heap memory capacity includes the following steps 1 to 4:

[0166] Step 1: Obtain an upper boundary and a lower boundary of the heap memory capacity according to the original heap memory capacity.

[0167] Step 2: normalize the target heap memory capacity based on the upper boundary and the lower boundary to obtain a normalized value of the target heap memory capacity.

[0168] In some embodiments, normalizing the target heap memory capacity based on the upper boundary and the lower boundary includes: normalizing the target heap memory capacity based on a sigmoid function, the upper boundary, and the lower boundary.

[0169] Exemplarily, the process of obtaining the normalized value of the target heap memory capacity is described below by taking the normalization function as a sigmoid function, the upper boundary as 13, the lower boundary as 7, and the target heap memory capacity as 15 as an example.

[0170] First, map 15 to the interval [0,1] to obtain: (15-7) / (13-7)=4 / 3.

[0171] Secondly, substituting 4 / 3 into the Sigmoid function yields:

[0172]

[0173] Exemplarily, the process of obtaining the normalized value of the target heap memory capacity is described below by taking the normalization function as a sigmoid function, the upper boundary as 13, the lower boundary as 7, and the target heap memory capacity as 12 as an example.

[0174] First, map 12 to the interval [0,1] to obtain: (12-7) / (13-7)=5 / 6.

[0175] Secondly, substituting 5 / 6 into the Sigmoid function yields:

[0176]

[0177] Step 3: Obtain a denormalized value of the target heap memory capacity according to the normalized value, the upper boundary, and the lower boundary.

[0178] As shown in the above example, when the upper boundary is 13, the lower boundary is 7, and the target heap memory capacity is 15, the normalized value is 0.908, and the inverse normalized value of the target heap memory capacity obtained according to the normalized value, the upper boundary and the lower boundary is: 7+0.908*(13-7)=12.448.

[0179] As shown in the above example, when the upper boundary is 13, the lower boundary is 7, and the target heap memory capacity is 15, the normalized value is 0.885, and the inverse normalized value of the target heap memory capacity obtained according to the normalized value, the upper boundary and the lower boundary is: 7+0.885*(13-7)=12.31.

[0180] Step 4: configure the denormalized value as the capacity of the heap memory.

[0181] The above embodiment obtains the upper and lower boundaries of the heap memory capacity according to the original heap memory capacity, and then normalizes the target heap memory capacity based on the upper and lower boundaries to obtain a normalized value of the target heap memory capacity, and obtains an inverse normalized value of the target heap memory capacity according to the normalized value, the upper and lower boundaries, and configures the inverse normalized value as the capacity of the heap memory. Therefore, the above embodiment can also make the configured heap memory capacity always within the upper and lower boundaries of the heap memory capacity, thereby avoiding excessive adjustment of the heap memory capacity, which in turn affects the accuracy of object promotion rate estimation.

[0182] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a memory management device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the memory management device in this embodiment can correspond to all the contents of the above method embodiment.

[0183] The present application embodiment provides a memory management device, Figure 3 is a structural diagram of the memory management device, such as Figure 3 As shown, the memory management device 300 includes:

[0184] An acquisition unit 31 is used to acquire log data of garbage collection performed on heap memory;

[0185] A parsing unit 32, configured to obtain an indicator related to the memory usage of the old generation according to the log data;

[0186] A construction unit 33 is used to construct a memory estimation model with full heap garbage collection frequency as an independent variable and estimated heap memory capacity as a dependent variable according to the indicator related to the old generation memory usage and the original heap memory capacity of the heap memory;

[0187] A processing unit 34 is used to obtain a target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency;

[0188] The configuration unit 35 is used to configure the capacity of the heap memory according to the target heap memory capacity.

[0189] As an optional implementation of the embodiment of the present application, the parsing unit 32 is specifically used to obtain the historical promotion rate, the historical application rate and the historical survival capacity according to the log data;

[0190] Among them, the historical promotion rate is the rate at which young generation memory is promoted to old generation memory; the historical application rate is the application rate of old generation memory; and the historical survival capacity is the size of surviving objects after full heap garbage collection.

[0191] As an optional implementation of the embodiment of the present application, the construction unit 33 is specifically used to construct a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate based on the original heap memory capacity and the historical promotion rate; construct a second conversion relationship between the full heap garbage collection frequency and the estimated heap memory capacity based on the historical survival capacity, the historical application rate, and the estimated promotion rate; and construct the memory estimation model based on the first conversion relationship and the second conversion relationship.

[0192] As an optional implementation of the embodiment of the present application, the construction unit 33 is specifically used to obtain a first parameter based on a change in the estimated heap memory capacity relative to the original heap memory capacity; and compensate the historical promotion rate based on the first parameter to obtain the first conversion relationship.

[0193] As an optional implementation of the embodiment of the present application, the construction unit 33 is specifically used to obtain the first old generation memory usage based on the historical application rate and the full-heap garbage collection frequency; obtain the second old generation memory usage based on the estimated promotion rate and the full-heap garbage collection frequency; construct a conversion relationship between the old generation memory usage and the full-heap garbage collection frequency based on the first old generation memory usage, the second old generation memory usage and the historical surviving capacity; generate the second conversion relationship based on the conversion relationship between the old generation memory usage and the full-heap garbage collection frequency.

[0194] As an optional implementation of the embodiment of the present application, the configuration unit 35 is specifically used to obtain the upper and lower boundaries of the heap memory capacity based on the original heap memory capacity; determine whether the target heap memory capacity is greater than the lower boundary and less than the upper boundary; if the target heap memory capacity is greater than the lower boundary and less than the upper boundary, configure the target heap memory capacity as the capacity of the heap memory; if the target heap memory capacity is less than or equal to the lower boundary, configure the lower boundary as the capacity of the heap memory; if the target heap memory capacity is greater than or equal to the upper boundary, configure the upper boundary as the capacity of the heap memory.

[0195] As an optional implementation of the embodiment of the present application, the configuration unit 35 is specifically used to obtain the upper boundary and the lower boundary of the heap memory capacity based on the original heap memory capacity; normalize the target heap memory capacity based on the upper boundary and the lower boundary to obtain the normalized value of the target heap memory capacity; obtain the inverse normalized value of the target heap memory capacity based on the normalized value, the upper boundary and the lower boundary; and configure the inverse normalized value as the capacity of the heap memory.

[0196] As an optional implementation of an embodiment of the present application, the parsing unit 32 is specifically used to parse the log data to obtain historical garbage collection events; use the K-Means classification algorithm to cluster the historical garbage collection events into two clusters based on the frequency of occurrence; and obtain indicators related to the old generation memory usage through the historical garbage collection events in the cluster with a higher occurrence frequency of the two clusters.

[0197] As an optional implementation of the embodiment of the present application, the acquisition unit 31 is specifically used to obtain log data of garbage collection executed by the JVM in the stream processing business system.

[0198] The memory management device provided in the embodiment of the present application can execute the memory management method provided in any of the above embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0199] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device provided in this embodiment includes: a memory 401 and a processor 402, wherein the memory 401 is used to store a computer program, and the processor 402 is used to execute any memory management method provided in the above embodiments when executing the computer program.

[0200] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements any memory management method provided in the above embodiments.

[0201] Based on the same inventive concept, an embodiment of the present application further provides a computer program product. When the computer program product is executed on a computer, the computing device implements any memory management method provided in the above embodiments.

[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0203] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0204] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0205] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A memory management method, characterized in that: include: Get the log data of garbage collection performed on heap memory; Obtaining indicators related to old generation memory usage according to the log data; According to the indicators related to the old generation memory usage and the original heap memory capacity of the heap memory, a memory estimation model is constructed with the full heap garbage collection frequency as an independent variable and the estimated heap memory capacity as a dependent variable; Obtaining a target heap memory capacity according to the memory estimation model and the expected full heap garbage collection frequency; The capacity of the heap memory is configured according to the target heap memory capacity.

2. The method according to claim 1, characterized in that The obtaining of indicators related to the memory usage of the old generation according to the log data includes: Obtaining historical promotion rate, historical application rate and historical survival capacity according to the log data; Among them, the historical promotion rate is the rate at which the young generation memory is promoted to the old generation memory; the historical application rate is the application rate of the old generation memory; and the historical survival capacity is the size of the surviving objects after full heap garbage collection.

3. The method according to claim 2, characterized in that The step of constructing a memory estimation model with full heap garbage collection frequency as an independent variable and estimated heap memory capacity as a dependent variable based on the index related to the old generation memory usage and the original heap memory capacity of the heap memory includes: Constructing a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate according to the original capacity and the historical promotion rate; Constructing a second conversion relationship between the full-heap garbage collection frequency and the estimated heap memory capacity according to the historical surviving capacity, the historical application rate, and the estimated promotion rate; The memory estimation model is constructed according to the first conversion relationship and the second conversion relationship.

4. The method according to claim 3, characterized in that The constructing a first conversion relationship between the estimated heap memory capacity and the estimated promotion rate according to the original heap memory capacity and the historical promotion rate includes: Acquire a first parameter according to a change in the estimated heap memory capacity relative to the original heap memory capacity; The historical promotion rate is compensated according to the first parameter to obtain the first conversion relationship.

5. The method according to claim 3, characterized in that: The constructing a second conversion relationship between the full heap garbage collection frequency and the estimated heap memory capacity according to the historical surviving capacity, the historical application rate, and the estimated promotion rate includes: Obtaining the memory usage of the first old generation according to the historical application rate and the full-heap garbage collection frequency; Obtaining the memory usage of the second old generation according to the estimated promotion rate and the full-heap garbage collection frequency; Constructing a conversion relationship between the old generation memory usage and the full heap garbage collection frequency according to the first old generation memory usage, the second old generation memory usage, and the historical surviving capacity; The second conversion relationship is generated according to the conversion relationship between the old generation memory usage and the full heap garbage collection frequency.

6. The method according to claim 1, characterized in that The configuring the capacity of the heap memory according to the target heap memory capacity includes: Acquire an upper boundary and a lower boundary of the heap memory capacity according to the original heap memory capacity; Determine whether the target heap memory capacity is greater than the lower limit and less than the upper limit; If the target heap memory capacity is greater than the lower limit and less than the upper limit, configuring the target heap memory capacity as the capacity of the heap memory; If the target heap memory capacity is less than or equal to the lower limit, configuring the lower limit as the capacity of the heap memory; If the target heap memory capacity is greater than or equal to the upper limit, the upper limit is configured as the capacity of the heap memory.

7. The method according to claim 1, characterized in that The configuring the capacity of the heap memory according to the target heap memory capacity includes: Acquire an upper boundary and a lower boundary of the heap memory capacity according to the original heap memory capacity; Normalizing the target heap memory capacity based on the upper boundary and the lower boundary to obtain a normalized value of the target heap memory capacity; Obtaining a denormalized value of the target heap memory capacity according to the normalized value, the upper boundary, and the lower boundary; The denormalized value is configured as the capacity of the heap memory.

8. The method according to claim 1, characterized in that The obtaining of indicators related to the memory usage of the old generation according to the log data includes: Parsing the log data to obtain historical garbage collection events; Using the K-Means classification algorithm, the historical garbage collection events are clustered into two clusters based on the frequency of occurrence; An indicator related to the memory usage of the old generation is obtained through historical garbage collection events in the cluster with a higher occurrence frequency among the two clusters.

9. The method according to claim 1, characterized in that: The obtaining of log data of garbage collection performed on the heap memory includes: Obtain log data of garbage collection performed by the JVM in the stream processing business system.

10. A memory management device, characterized in that: include: An acquisition unit, used for acquiring log data of garbage collection performed on heap memory; A parsing unit, used to obtain indicators related to the memory usage of the old generation according to the log data; A construction unit, configured to construct a memory estimation model with full heap garbage collection frequency as an independent variable and estimated heap memory capacity as a dependent variable according to an indicator related to the old generation memory usage and an original heap memory capacity of the heap memory; A processing unit, configured to obtain a target heap memory capacity according to the memory estimation model and an expected full heap garbage collection frequency; A configuration unit is used to configure the capacity of the heap memory according to the target heap memory capacity.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the electronic device to implement the memory management method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the memory management method according to any one of claims 1 to 9.

13. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to implement the memory management method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Method for adjusting maximum heap memory of virtual machine and electronic equipment

    CN120743437A