A memory adjustment method and terminal

By obtaining and analyzing the GC logs of the task node, the estimated memory size of the data processing task is calculated, and resources are dynamically adjusted, which solves the problem of invalid occupation of resources and maximizes resource usage efficiency.

CN113918335BActive Publication Date: 2024-11-22FUJIAN TIANQUAN EDUCATION TECH LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111195973.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-14
Publication Date
2024-11-22
Estimated Expiration
2041-10-14

AI Technical Summary

Technical Problem

The prior art often estimates large resource values ​​in data processing tasks, resulting in invalid occupation of resources and failing to maximize resource usage efficiency.

Method used

By obtaining the GC log of the task to be adjusted, obtaining its unique identifier on each task node and the maximum heap memory size after Full GC, calculate the estimated memory size, and perform memory adjustments to dynamically adjust resource usage.

Benefits of technology

This avoids the estimated memory situation, reduces the use of invalid resources, and maximizes resource utilization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113918335B_ABST
    Figure CN113918335B_ABST
Patent Text Reader

Abstract

The present invention discloses a memory adjustment method and a terminal, which obtain a task to be adjusted, asynchronously collect the GC log of each task node of the task to be adjusted; obtain the unique identifier of the task to be adjusted and the maximum heap memory size of the task to be adjusted after Full GC on each task node from the GC log, and then calculate the estimated memory size of the task to be adjusted, which can avoid the situation where the estimated memory is too small; therefore, the heap memory size information in the GC log is automatically analyzed to calculate the number of resources required by the task to be adjusted, and dynamically adjust the memory, which can reduce the occupation of invalid resources, thereby maximizing the resource utilization efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a memory adjustment method and a terminal. Background Art

[0002] With the booming development of mobile Internet, there are various client apps, and each app needs corresponding server support. These large numbers of apps bring users, and the user data also increases rapidly. Due to the increase in data volume, various online / offline task processing also brings great pressure.

[0003] At present, for various data processing tasks, resources are generally estimated. For example, a data calculation task is estimated to use a fixed memory for data processing. Or if the machine resources are sufficient, each task directly applies for resources according to the maximum amount of resources to ensure that the task can be completed normally. However, in general, we will estimate a larger resource value so that the task can be fully processed. Therefore, the estimated value will be much different from the actual value. In this way, more resources are estimated, but the actual resources used by the task are not so much, which leads to invalid occupation of resources and failure to maximize resource utilization efficiency. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a memory adjustment method and a terminal, which can reduce the occupation of invalid resources in the memory and improve the resource utilization rate.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A memory adjustment method comprises the steps of:

[0007] Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted;

[0008] Obtaining, from the GC log, a unique identifier of the task to be adjusted and a maximum heap memory size of the task to be adjusted after Full GC on each task node;

[0009] The estimated memory size of the task to be adjusted is calculated according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and the memory is adjusted.

[0010] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0011] A memory adjustment terminal comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0012] Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted;

[0013] Obtaining, from the GC log, a unique identifier of the task to be adjusted and a maximum heap memory size of the task to be adjusted after Full GC on each task node;

[0014] The estimated memory size of the task to be adjusted is calculated according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and the memory is adjusted.

[0015] The beneficial effects of the present invention are: obtaining the task to be adjusted, asynchronously collecting the GC log of each task node of the task to be adjusted; obtaining the unique identifier of the task to be adjusted and the maximum heap memory size of the task to be adjusted after Full GC on each task node from the GC log, and then calculating the estimated memory size of the task to be adjusted, thereby avoiding the situation where the estimated memory is too small; therefore, automatically analyzing the heap memory size information in the GC log, calculating the amount of resources required for the task to be adjusted, and dynamically adjusting the memory, thereby reducing the occupation of invalid resources, thereby maximizing the resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a memory adjustment method according to an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a memory adjustment terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0019] Please refer to Figure 1 , an embodiment of the present invention provides a memory adjustment method, comprising the steps of:

[0020] Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted;

[0021] Obtaining, from the GC log, a unique identifier of the task to be adjusted and a maximum heap memory size of the task to be adjusted after Full GC on each task node;

[0022] The estimated memory size of the task to be adjusted is calculated according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and the memory is adjusted.

[0023] From the above description, it can be seen that the beneficial effects of the present invention are: obtaining the task to be adjusted, asynchronously collecting the GC log of each task node of the task to be adjusted; obtaining the unique identifier of the task to be adjusted and the maximum heap memory size of the task to be adjusted after Full GC on each task node from the GC log, and then calculating the estimated memory size of the task to be adjusted, which can avoid the situation where the estimated memory is too small; therefore, the heap memory size information in the GC log is automatically analyzed to calculate the number of resources required for the task to be adjusted, and dynamically adjust the memory, which can reduce the occupation of invalid resources, thereby maximizing the resource utilization efficiency.

[0024] Furthermore, obtaining the unique identifier of the task to be adjusted from the GC log, and the maximum heap memory size of the task to be adjusted after Full GC on each task node includes:

[0025] Obtaining a unique identifier of the task to be adjusted according to the GC log, and recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier;

[0026] Calculate the maximum heap memory size of each task node of the task to be adjusted after Full GC.

[0027] From the above description, we can know that by recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier of the task to be adjusted, we can count the actual memory usage of each task subnode after each Full GC during its task cycle, and select the maximum memory usage as the standard, so as to quickly and conveniently obtain the key memory information after each Full GC, which is convenient for the subsequent calculation of the estimated memory size.

[0028] Furthermore, the estimated memory size of the task to be adjusted is calculated based on the maximum heap memory size of the task to be adjusted after Full GC on each task node, including:

[0029] Accumulate the maximum heap memory size of each task node of the task to be adjusted after Full GC to obtain the maximum heap memory size of the task to be adjusted after Full GC;

[0030] The adjustment coefficient is obtained through stress testing practice, and the estimated memory size of the task to be adjusted is obtained by multiplying the maximum heap memory size after Full GC of the adjustment task by the adjustment coefficient.

[0031] From the above description, we can know that the maximum heap memory size after Full GC of each task node is accumulated to obtain the maximum heap memory size after Full GC of the task to be adjusted, and the adjustment coefficient is obtained through stress testing practice, so as to calculate the estimated memory size of the task to be adjusted, avoid the situation where the calculated estimated memory is too small, and ensure the accuracy of the estimated memory calculation.

[0032] Furthermore, the memory adjustment includes:

[0033] Determine whether the estimated memory size is smaller than the initial memory of the task to be adjusted; if so, reduce the initial memory to the estimated memory size; otherwise, increase the initial memory to the estimated memory size.

[0034] From the above description, it can be seen that the estimated memory size is smaller than the initial memory of the task to be adjusted. Reducing the initial memory to the estimated memory size can reduce the occupation of invalid resources; conversely, increasing the initial memory to the estimated memory size can avoid insufficient memory.

[0035] Further, the asynchronously collecting the GC log of each task node of the task to be adjusted includes:

[0036] When the task to be adjusted is executed, asynchronously collecting the GC log of each task node of the task to be adjusted;

[0037] The memory adjustment includes:

[0038] Restarting the task to be adjusted, the memory of the task to be adjusted being the estimated memory size.

[0039] From the above description, it can be seen that when the task to be adjusted is executed, the GC log of each task node of the task to be adjusted is collected asynchronously, so that the GC log can be collected without affecting the task to be adjusted. After the memory is adjusted, when the task to be adjusted is started again, the estimated memory size can take effect dynamically, thereby improving resource utilization.

[0040] Please refer to Figure 2 Another embodiment of the present invention provides a memory adjustment terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0041] Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted;

[0042] Obtaining, from the GC log, a unique identifier of the task to be adjusted and a maximum heap memory size of the task to be adjusted after Full GC on each task node;

[0043] The estimated memory size of the task to be adjusted is calculated according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and the memory is adjusted.

[0044] From the above description, we can know that we obtain the task to be adjusted and asynchronously collect the GC log of each task node of the task to be adjusted; obtain the unique identifier of the task to be adjusted and the maximum heap memory size of the task to be adjusted after FullGC on each task node from the GC log, and then calculate the estimated memory size of the task to be adjusted, which can avoid the situation where the estimated memory is too small; therefore, we automatically analyze the heap memory size information in the GC log, calculate the number of resources required by the task to be adjusted, and dynamically adjust the memory, which can reduce the occupation of invalid resources and maximize the efficiency of resource utilization.

[0045] Furthermore, obtaining the unique identifier of the task to be adjusted from the GC log, and the maximum heap memory size of the task to be adjusted after Full GC on each task node include:

[0046] Obtaining a unique identifier of the task to be adjusted according to the GC log, and recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier;

[0047] Calculate the maximum heap memory size of each task node of the task to be adjusted after Full GC.

[0048] From the above description, we can know that by recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier of the task to be adjusted, we can count the actual memory usage of each task subnode after each Full GC during its task cycle, and select the maximum memory usage as the standard, so as to quickly and conveniently obtain the key memory information after each Full GC, which is convenient for the subsequent calculation of the estimated memory size.

[0049] Furthermore, the estimated memory size of the task to be adjusted is calculated based on the maximum heap memory size of the task to be adjusted after Full GC on each task node, including:

[0050] Accumulate the maximum heap memory size of each task node of the task to be adjusted after Full GC to obtain the maximum heap memory size of the task to be adjusted after Full GC;

[0051] The adjustment coefficient is obtained through stress testing practice, and the estimated memory size of the task to be adjusted is obtained by multiplying the maximum heap memory size after Full GC of the adjustment task by the adjustment coefficient.

[0052] From the above description, we can know that the maximum heap memory size after Full GC of each task node is accumulated to obtain the maximum heap memory size after Full GC of the task to be adjusted, and the adjustment coefficient is obtained through stress testing practice, so as to calculate the estimated memory size of the task to be adjusted, avoid the situation where the calculated estimated memory is too small, and ensure the accuracy of the estimated memory calculation.

[0053] Furthermore, the memory adjustment includes:

[0054] Determine whether the estimated memory size is smaller than the initial memory of the task to be adjusted; if so, reduce the initial memory to the estimated memory size; otherwise, increase the initial memory to the estimated memory size.

[0055] From the above description, it can be seen that the estimated memory size is smaller than the initial memory of the task to be adjusted. Reducing the initial memory to the estimated memory size can reduce the occupation of invalid resources; conversely, increasing the initial memory to the estimated memory size can avoid insufficient memory.

[0056] Further, the asynchronously collecting the GC log of each task node of the task to be adjusted includes:

[0057] When the task to be adjusted is executed, asynchronously collecting the GC log of each task node of the task to be adjusted;

[0058] The memory adjustment includes:

[0059] Restarting the task to be adjusted, the memory of the task to be adjusted being the estimated memory size.

[0060] From the above description, it can be seen that when the task to be adjusted is executed, the GC log of each task node of the task to be adjusted is collected asynchronously, so that the GC log can be collected without affecting the task to be adjusted. After the memory is adjusted, when the task to be adjusted is started again, the estimated memory size can take effect dynamically, thereby improving resource utilization.

[0061] A memory adjustment method and terminal of the present invention are suitable for dynamically adjusting the memory of distributed tasks, reducing the occupation of invalid resources, and maximizing the resource utilization efficiency. The following is an explanation through specific implementation methods:

[0062] Embodiment 1

[0063] Please refer to Figure 1, a memory adjustment method, comprising the steps of:

[0064] S1. Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted.

[0065] When the task to be adjusted is executed, the GC log of each task node of the task to be adjusted is collected asynchronously.

[0066] Specifically, in this embodiment, the task to be adjusted is a real-time data statistics task, and the estimated memory usage before execution of the task is 8G;

[0067] In software using the Java language series, there are related GC logs during the task running process. There is an asynchronous task that can collect and analyze GC logs from each task node.

[0068] S2. Obtain the unique identifier of the task to be adjusted from the GC log, and the maximum heap memory size of the task to be adjusted after Full GC on each task node.

[0069] Step S2 includes:

[0070] Obtaining a unique identifier of the task to be adjusted according to the GC log, and recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier;

[0071] Calculate the maximum heap memory size of each task node of the task to be adjusted after Full GC.

[0072] Specifically, in this embodiment, there is an asynchronous task that can collect and analyze GC logs from each task node, where the key information collected includes taskId and related information before and after each Full GC, that is, the memory heap size before Full GC and the memory heap size after Full GC.

[0073] The analysis program can record the heap memory size of a task after Full GC on different task nodes according to taskId. At this time, the maximum heap memory size of the same task after Full GC on each task node can be obtained.

[0074] S3. Calculate the estimated memory size of the task to be adjusted according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and perform memory adjustment.

[0075] S31, accumulating the maximum heap memory size of each task node of the task to be adjusted after Full GC to obtain the maximum heap memory size of the task to be adjusted after Full GC;

[0076] The adjustment coefficient is obtained through stress testing practice, and the estimated memory size of the task to be adjusted is obtained by multiplying the maximum heap memory size after Full GC of the adjustment task by the adjustment coefficient.

[0077] Specifically, add up the maximum heap memory size of each task node to get the maximum heap memory size X after Full GC of the task;

[0078] According to the analysis of historical task GC logs and repeated stress testing practices, the adjustment coefficient is obtained. In this implementation, the adjustment coefficient is 4. At this time, the optimal estimated memory size of the task Y=4X can be known. The adjustment coefficient varies depending on the task to be adjusted and its operating environment.

[0079] S32, determining whether the estimated memory size is smaller than the initial memory of the task to be adjusted, if so, reducing the initial memory to the estimated memory size, otherwise, increasing the initial memory to the estimated memory size.

[0080] Specifically, if Y is smaller than the initial memory of the previous task, the task resource estimation module needs to be dynamically notified, and the initial memory of the task will be lowered next time. If Y is larger than the initial memory of the previous task, it means that the historical initial memory size may have hidden dangers and there may be a possibility of insufficient memory usage, and the initial memory size of the task needs to be appropriately adjusted.

[0081] S4. Restarting the task to be adjusted, the memory of the task to be adjusted being the estimated memory size, that is, when the task to be adjusted is restarted, the estimated memory size can take effect dynamically.

[0082] Therefore, this embodiment automatically analyzes key memory information and can ultimately calculate the number of resources required for a task, thereby dynamically adjusting the estimated resource value of the task, thereby reducing the occupancy of invalid resources and maximizing resource utilization efficiency.

[0083] Embodiment 2

[0084] Please refer to Figure 2 A memory adjustment terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of a memory adjustment method of embodiment 1 is implemented.

[0085] In summary, the present invention provides a memory adjustment method and terminal, which obtains the task to be adjusted and asynchronously collects the GC log of each task node of the task to be adjusted; records the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier of the task to be adjusted, and can count the actual memory usage of each task subnode after each Full GC in its task cycle, and select the maximum memory usage as the standard, so as to quickly and conveniently obtain the key memory information after each Full GC; accumulates the maximum heap memory size of each task node after Full GC to obtain the Full GC size of the task to be adjusted. The maximum heap memory size after GC is obtained, and the adjustment coefficient is obtained through stress testing practice, so as to calculate the estimated memory size of the task to be adjusted, and avoid the situation where the calculated estimated memory is too small; and when the task to be adjusted is executed, the GC log of each task node of the task to be adjusted is collected asynchronously, so that the GC log can be collected without affecting the task to be adjusted. After the memory is adjusted, when the task to be adjusted is started again, the estimated memory size can take effect dynamically, thereby improving resource utilization. Therefore, the heap memory size information in the GC log is automatically analyzed, the number of resources required by the task to be adjusted is calculated, and the memory is adjusted dynamically, which can reduce the occupation of invalid resources and maximize resource utilization efficiency.

[0086] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A memory adjustment method, characterized in that: Includes steps: Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted; Obtaining, from the GC log, a unique identifier of the task to be adjusted and a maximum heap memory size of the task to be adjusted after Full GC on each task node; The estimated memory size of the task to be adjusted is calculated according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and the memory is adjusted.

2. A memory adjustment method according to claim 1, characterized in that: The unique identifier of the task to be adjusted is obtained from the GC log, and the maximum heap memory size of the task to be adjusted after Full GC on each task node includes: Obtaining a unique identifier of the task to be adjusted according to the GC log, and recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier; Calculate the maximum heap memory size of each task node of the task to be adjusted after Full GC.

3. A memory adjustment method according to claim 1, characterized in that: The estimated memory size of the task to be adjusted is calculated based on the maximum heap memory size of the task to be adjusted after Full GC on each task node, including: Accumulate the maximum heap memory size of each task node of the task to be adjusted after Full GC to obtain the maximum heap memory size of the task to be adjusted after Full GC; The adjustment coefficient is obtained through stress testing practice, and the estimated memory size of the task to be adjusted is obtained by multiplying the maximum heap memory size after Full GC of the adjustment task by the adjustment coefficient.

4. A memory adjustment method according to claim 1, characterized in that: The memory adjustment comprises: Determine whether the estimated memory size is smaller than the initial memory of the task to be adjusted; if so, reduce the initial memory to the estimated memory size; otherwise, increase the initial memory to the estimated memory size.

5. A memory adjustment method according to any one of claims 1 to 4, characterized in that: The asynchronously collecting the GC log of each task node of the task to be adjusted includes: When the task to be adjusted is executed, asynchronously collecting the GC log of each task node of the task to be adjusted; The memory adjustment includes: Restarting the task to be adjusted, the memory of the task to be adjusted being the estimated memory size.

6. A memory adjustment terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Obtain the task to be adjusted, and asynchronously collect the GC log of each task node of the task to be adjusted; Obtaining, from the GC log, a unique identifier of the task to be adjusted and a maximum heap memory size of the task to be adjusted after Full GC on each task node; The estimated memory size of the task to be adjusted is calculated according to the maximum heap memory size after Full GC of the task to be adjusted on each task node, and the memory is adjusted.

7. A memory adjustment terminal according to claim 6, characterized in that: The unique identifier of the task to be adjusted is obtained from the GC log, and the maximum heap memory size of the task to be adjusted after Full GC on each task node includes: Obtaining a unique identifier of the task to be adjusted according to the GC log, and recording the heap memory size data of each task node of the task to be adjusted before and after Full GC according to the unique identifier; Calculate the maximum heap memory size of each task node of the task to be adjusted after Full GC.

8. A memory adjustment terminal according to claim 6, characterized in that: The estimated memory size of the task to be adjusted is calculated based on the maximum heap memory size of the task to be adjusted after Full GC on each task node, including: Accumulate the maximum heap memory size of each task node of the task to be adjusted after Full GC to obtain the maximum heap memory size of the task to be adjusted after Full GC; The adjustment coefficient is obtained through stress testing practice, and the estimated memory size of the task to be adjusted is obtained by multiplying the maximum heap memory size after Full GC of the adjustment task by the adjustment coefficient.

9. A memory adjustment terminal according to claim 6, characterized in that: The memory adjustment comprises: Determine whether the estimated memory size is smaller than the initial memory of the task to be adjusted; if so, reduce the initial memory to the estimated memory size; otherwise, increase the initial memory to the estimated memory size.

10. A memory adjustment terminal according to any one of claims 6 to 9, characterized in that: The asynchronously collecting the GC log of each task node of the task to be adjusted includes: When the task to be adjusted is executed, asynchronously collecting the GC log of each task node of the task to be adjusted; The memory adjustment includes: Restarting the task to be adjusted, the memory of the task to be adjusted being the estimated memory size.