Method for dynamically updating memory resource application
By introducing a method of dynamic update memory resource application in Hadoop YARN, the problem of poor resource utilization caused by traditional static resource allocation strategies is solved, dynamic adjustment and optimization of memory resources are achieved, and task execution efficiency and system reliability are improved.
Patent Information
- Application Number
- CN202510118004.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-24
- Publication Date
- 2025-05-27
AI Technical Summary
Traditional static resource allocation strategies lead to poor resource utilization in big data processing and the inability to dynamically adjust memory resources, resulting in resource waste and task delay.
By introducing a method of dynamically updating memory resource applications in Hadoop YARN, the resource scheduler and task manager are used to regularly monitor the actual memory usage of the application, dynamically adjust resource requests, and realize dynamic recycling and new applications of memory resources.
It improves resource utilization, reduces memory resource waste, optimizes task execution speed and responsiveness, adapts to the application's dynamic needs for memory resources, and reduces operation and maintenance complexity.
Smart Images
Figure CN120045322A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of big data processing in cloud computing, and more specifically, to a method for dynamically updating memory resource applications in big data processing technology. Background Art
[0002] With the explosive growth of data volume, traditional centralized computing models are unable to effectively process large-scale data sets. Distributed computing provides a solution by distributing tasks across multiple nodes to improve computing power and data processing efficiency. In a distributed environment, resource management and job scheduling are two major challenges. In the field of big data processing, Hadoop YARN (Yet Another Resource Negotiator), as the core component of resource management and scheduling, plays a crucial role. Traditional resource allocation strategies usually rely on static (fixed) configurations, that is, applications pre-define memory and computing resources at startup. However, with the continuous increase in data scale and computing complexity, this static resource management method faces significant limitations.
[0003] Firstly, static resource allocation may lead to poor resource utilization. When the load of an application fluctuates during execution, the pre-allocated fixed resources may not meet peak demands, or may cause resource waste during low-load periods. This not only affects the overall efficiency of the cluster but may also lead to task delays when resources are strained.
[0004] Secondly, modern applications (such as stream processing and interactive analysis) require higher flexibility and response speed, which makes dynamic resource management necessary. Dynamic memory resource applications allow applications to adjust their resource occupancy during runtime based on real-time load and performance monitoring. This mechanism can not only optimize resource utilization but also improve the execution speed and response ability of tasks.
[0005] Real-time monitoring technology provides the technical basis for introducing dynamic memory management to YARN. By integrating advanced monitoring tools and dynamic update algorithms, YARN can detect changes in the resource requirements of applications in real time and dynamically adjust resource applications and allocations to ensure optimized resource usage without sacrificing performance.
[0006] This ability to dynamically improve memory resource applications is crucial for enhancing the efficiency and adaptability of Hadoop clusters, especially in computing environments with high resource volatility and the need for quick response. By applying this technology to YARN, a more efficient and economical resource management strategy can be achieved on big data processing platforms.
[0007] In view of the above problems, the present invention proposes a solution. Summary of the Invention
[0008] To overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a method for dynamically updating memory resource requests, which solves the problem of low resource utilization rate caused by fixed memory resource requests during the operation of distributed parallel tasks in Hadoop Yarn. That is, after the task initially submits a request for memory resources, it cannot dynamically adjust the memory request according to the actual memory resource usage, resulting in waste of memory resources.
[0009] To achieve the above object, the present invention provides the following technical solutions:
[0010] A method for dynamically updating memory resource requests includes the following steps:
[0011] Submit an application request, use the resource scheduler in Yarn to process the application request, and start the task manager; request to allocate the required memory resources, the resource scheduler accepts the request and finds a machine that meets the resource requirements, and allocates the resource request through the resource pool; during the task execution, regularly obtain the actual memory resource usage of the application and update the actual memory resource quantity in the resource usage table; check the size relationship between the actual memory resource quantity and the required memory resource quantity for the application through the resource monitoring program, and update the resource request according to the size relationship between the actual memory resource quantity and the required memory resource quantity; when the application ends, release the resources and update the resource usage table and the machine resource pool table.
[0012] In a preferred embodiment, the process of submitting the application request is as follows: when the client has an application to run, it will prepare the relevant information of the application, including the memory resources required for the application to run.
[0013] In a preferred embodiment, the process of using the resource scheduler in Yarn to process the application request is as follows: after receiving the application request submitted by the client, parse and verify the request; the parsing and verification of the request include checking the format of the request, various parameters, and the memory resource quantity declared by the client; and plan how to allocate resources for the application according to the scheduling algorithm of the resource scheduler and the resource usage status of the system.
[0014] In a preferred embodiment, the process of starting the task manager is as follows: the resource scheduler starts the task manager; the task manager submits a specific resource request to YARN and interacts frequently with the resource management module of YARN; convey the specific information of the memory resources required by the application so that YARN can allocate resources for the application more accurately; after the resource allocation is completed, the task manager monitors the resource usage of the application during the running process and feeds back to the resource scheduler.
[0015] In a preferred embodiment, the process of the request allocating the required memory resources and the resource scheduler accepting the request and finding a machine that meets the resource requirements is as follows: The client requests to allocate the required memory resources, and the resource scheduler in YARN accepts the request and finds a machine that meets the resource requirements; the machine that meets the resource requirements includes a machine whose available resources are greater than or equal to the requested resources; the initial available resources of the machine are the maximum available resources configured for the machine.
[0016] In a preferred embodiment, the process of allocating resource requests through a resource pool is as follows: After allocating resources, the state of the resource pool changes, and the available resources change from the original T to T - Rm, where T represents the initial available resources, Rm represents the allocated resources, and T - Rm represents the remaining available resources; the allocated resources change from 0 to Rm, and the requested allocated resources in the resource usage table change from 0 to Rm, where Rm represents the allocated resources.
[0017] In a preferred embodiment, the steps of periodically obtaining the actual memory resources used by an application and updating the actual memory resources used quantity in the resource usage table are as follows:
[0018] The resource monitor periodically starts to collect the memory usage data of the application.
[0019] The monitor obtains the currently actual used memory resources from each application.
[0020] The monitor records the amount of memory resources actually used by the application into the resource usage table, and the resource usage table of each application will be updated; and this process is repeated at regular time intervals.
[0021] In a preferred embodiment, the process of updating the resource request according to the size relationship between the actual used memory resources quantity and the required memory resources quantity for application is as follows:
[0022] If the actual used memory resources quantity is much less than the required memory resources quantity for application, the resource application update and recycling process is triggered.
[0023] If the actual used memory resources quantity is much greater than the required memory resources quantity for application, a new resource application is triggered.
[0024] In a preferred embodiment, if the actual amount of memory resources used is much smaller than the amount of memory resources required for the application, the process of triggering resource application update and recovery is as follows: When the actual resources used are much smaller than the resources applied for, resource recovery will be triggered. The resource recovery is carried out in batches according to a certain proportion; and the recovered resources are temporarily placed in the resource pool that can be used as an alternative by the machine; when the same task triggers resource recovery again next time, the resources recovered last time are put back into the available resource pool; while the resources recovered this time are still placed in the alternative available resource pool.
[0025] In a preferred embodiment, if the actual amount of memory resources used is much larger than the amount of memory resources required for the application, the process of triggering new resource application is as follows: When the actual resources used are greater than the resources applied for, a new resource application will be triggered, and it is judged whether there has been a situation of resource recovery before; and allocate from the alternative available resources. If the alternative available resources are not enough, allocate from the node available resources.
[0026] The technical effects and advantages of a method for dynamically updating memory resource applications according to the present invention:
[0027] 1. Different from the usual situation where the memory resources of a machine are divided into available memory resources and allocated resources, in the present invention, the memory resources of any machine in the resource pool are divided into three types: available memory resources, alternative available memory resources, and allocated resources. Among them, new task applications are all allocated from the available memory resources; this can improve the utilization rate of machine resources, reduce memory resource waste, and save costs; reduce the probability of task failure. When the memory application at the beginning of a task is insufficient, the amount of the application can be increased, avoiding the task being forcibly stopped by the resource monitoring process. Especially for some tasks with uneven data volume sizes and memory application peaks, their peak demands can be met, improving the performance of the application program.
[0028] 2. According to the actual memory resources used by the task, the present invention dynamically adjusts the memory resource application, can newly apply for or recover some memory resources, and the recovered resources can be used by other tasks. When a task has large fluctuations in the use of memory resources during operation, the resource utilization rate can be improved. It improves flexibility, can automatically adapt to the memory resource requirements of the program. In a multi-tenant environment, it avoids a large amount of resource reservation, avoids the situation where some node resources are idle while other node resources are tense. It improves the efficiency of program tuning. Users no longer need to worry about whether the manually set memory size meets the requirements, reduces the need for manual intervention, reduces the complexity of operation and maintenance, improves the user experience, and improves the reliability and availability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is the overall flowchart of a method for dynamically updating memory resource applications according to the present invention.
[0030] Figure 2 This is a schematic structural diagram of a method for dynamically updating memory resource applications according to the present invention.
[0031] Figure 3 It is the change process of the values in the resource usage table and the machine resource pool table after a task applies for resources.
[0032] Figure 4 It is the change process of the values in the resource usage table and the machine resource pool table after resource recovery at time t0.
[0033] Figure 5 It is the change process of the values in the resource usage table and the machine resource pool table after resource recovery at time t1.
[0034] Figure 6 It is the change process of the values in the resource usage table and the machine resource pool table during the resource new request stage.
[0035] Figure 7 It is the change process of the values in the resource usage table and the machine resource pool table during the resource new application stage.
[0036] Figure 8 It is the change process of the values in the resource usage table and the machine resource pool table when the application ends resource usage. Specific implementation manner
[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0038] Embodiment 1 Figure 1 It gives the overall flowchart of a method for dynamically updating memory resource applications according to the present invention.
[0039] Figure 2 It gives the schematic structural diagram of a method for dynamically updating memory resource applications according to the present invention.
[0040] S10. Submit an application request, where the request includes the required amount of memory resources, process the application request using the resource scheduler in Yarn, and start the task manager;
[0041] The client, as the initiator of the application, plays a key role in starting the entire process;
[0042] The process of submitting an application request is as follows:
[0043] When the client has an application that needs to run, it will prepare the relevant information of the application, including a very important item, which is to clarify the amount of memory resources required for the application to run.
[0044] The accurate setting of this value is crucial for the performance and stability of the application. Too much will cause waste of resources, and too little may cause the application to run incorrectly. The client integrates information such as the application code, configuration files, and memory resource requirements, and submits an application request to the YARN (Yet Another Resource Negotiator) system through specific interfaces and protocols.
[0045] The resource scheduler in YARN is like the brain of the entire resource management system, responsible for coordinating all resource requests.
[0046] The process of using the resource scheduler in yarn to process application requests is as follows:
[0047] When receiving the application request submitted by the client, it will first parse and verify the request, including: checking whether the format of the request is correct, whether all parameters are complete and reasonable, including whether the amount of memory resources declared by the client is within the range of resources that can be allocated by the system.
[0048] After confirming that the request is correct, the resource scheduler will start planning how to allocate resources for the application according to its own scheduling algorithm and the current resource usage status of the system. To further execute the resource allocation task, the resource scheduler will start the task manager.
[0049] The task manager is an important execution unit in the YARN architecture, and it works under the command of the resource scheduler.
[0050] The process of starting the task manager is as follows:
[0051] Once started by the resource scheduler, the task manager will immediately start processing the tasks assigned to it.
[0052] One of its core tasks is to submit specific resource requests to YARN. These requests are refined based on the amount of memory resources initially declared by the client and the preliminary allocation plan of the resource scheduler.
[0053] The task manager will interact frequently with the resource management module of YARN to accurately convey detailed information such as the specific specifications and usage duration of the memory resources required by the application, so that YARN can allocate resources for the application more precisely.
[0054] After the resource allocation is completed, the task manager is also responsible for monitoring the resource usage of the application during its operation, and promptly feedback to the resource scheduler to ensure that the entire execution process of the application is under effective resource control.
[0055] S20. Request to allocate the required memory resources. The resource scheduler accepts the request and searches for a machine that meets the resource requirements. The machine that meets the resource requirements allocates the resource request through the resource pool.
[0056] Apply for resources. When the client requests to allocate the required memory resources, the resource scheduler in YARN will accept this request and search for a machine that meets the resource requirements.
[0057] The machine that meets the resource requirements includes a machine whose available resources are greater than or equal to the requested resources.
[0058] Assume the allocated machine is Nodei. Suppose Nodei is allocated resources for the first time. Its initial available resources (Available) are the maximum available resources (T) configured for this machine, while Available_candidate is 0 and Allocated is 0.
[0059] Available_candidate being 0 means that there are no alternative available resources on the current machine, and Allocated being 0 means that no resources have been allocated to the machine.
[0060] The resource pool allocates the resource request.
[0061] After Nodei successfully allocates resources, the status of the resource pool on Nodei changes:
[0062] Available (available resources) decreases. It was originally T and now becomes T - Rm, indicating that Rm resources have been allocated, and the remaining available resources are T - Rm.
[0063] Allocated (allocated resources) changes from 0 to Rm, indicating that this machine has allocated Rm resources.
[0064] At the same time, Request in the resource usage table changes from 0 to Rm, indicating that the application has currently requested and allocated Rm resources. Figure 3 Shows the change process of the values in the resource usage table and the machine resource pool table after the task applies for resources.
[0065] S30. During the task execution, regularly obtain the actual memory resource usage of the application and update the actual memory resource quantity in the resource usage table.
[0066] During the task execution, resource monitoring will regularly track and update the actual memory usage of the application. The following are the specific steps:
[0067] Monitor startup: The resource monitor starts regularly (usually at time intervals) to collect the memory usage data of the application;
[0068] Obtain memory usage: The monitor obtains the currently actual used memory resources from each application; for example, it can access the monitoring data of the application to view the amount of memory consumed by the current application;
[0069] Update the resource usage table: The monitor records the amount of memory resources actually used by the application into the resource usage table; the resource usage table of each application will be updated, including the actually used memory (such as: Actual Memory Usage);
[0070] Regular update: This process will be repeated at certain time intervals to ensure that the resource monitoring information is up-to-date, so as to provide a basis for subsequent resource scheduling decisions.
[0071] Through regular monitoring and updating, the system can accurately master the actual memory resources used by each task (container), ensuring the efficiency and rationality of resource allocation.
[0072] S40, check the size relationship between the actually used memory resources quantity and the required memory resources quantity applied for through the resource monitoring program, and update the resource request according to the size relationship between the actually used memory resources quantity and the required memory resources quantity applied for;
[0073] If the actually used memory resources quantity U is much less than the required memory resources quantity Rm, that is, U + Bf < Rm (Bf is the reserved resource value applied for by the task), it will trigger the update and recycling process of the resource application;
[0074] At time t0, send a request to return resources. The calculation formula for the quantity of resources to be returned is as follows: Rt0 = (Rm - (U + Bf)) * R0; where, RO represents the recycling ratio each time, Rt0 is the quantity of resources to be returned, Rm is the required memory resources quantity applied for, U is the actually used memory resources quantity, and Bf is the reserved resource value applied for by the task;
[0075] The Return value in the resource usage table becomes Rt0, and the Request value becomes Rm - Rt0;
[0076] Where Return represents the memory resources returned in the previous round, and Request represents the memory applied for by the task (container);
[0077] The resource pool receives a return request and temporarily places it in Available_candidate. As a result, Available_candidate in the resource pool becomes Rt0;
[0078] Available_candidate represents the candidate available memory of the current node;
[0079] At this time, the resource pool does not directly place Rt0 into the Available resource because more resources may be needed in the future as the program runs; Figure 4 It shows the change process of the values in the resource usage table and the machine resource pool table after resource recovery at time t0.
[0080] At time t1, the resource monitor detects that the actual used memory resource quantity U continues to be less than the required memory resource quantity Rm for the application, that is, U + Bf < Rm,
[0081] Continue to return resources. The calculation formula for the quantity of the continued returned resources is as follows: Rt1 = (Rm - Rt0 - (U + Bf)) * RO;
[0082] In the formula, RO is the recycling ratio each time, Rt0 is the quantity of the returned resources sent in the previous round, Rm is the required memory resource quantity for the application, U is the actual used memory resource quantity, Bf is the reserved resource value for the task application, and Rt1 is the quantity of the continued returned resources;
[0083] Send a request to return Rt1 resources. When Nodei receives the resource request, it indicates that the Rt0 of the previous round can be moved from Available_candidate back to Available;
[0084] Available_candidate represents the candidate available memory of the current node;
[0085] At this time, in the resource usage table of the container, Request changes from Rm - Rt0 to Rm - Rt0 - Rt1, and Return changes from Rt0 to Rt1;
[0086] In the resource pool status table, Avaiable becomes T - Rm + Rt0, Available_candidate becomes Rt1, and Allocated becomes Rm - Rt0 - Rt1,
[0087] Similarly, at time ti, return Rti resources until Rm - U < Bf. Figure 5 It is the change process of the values in the resource usage table and the machine resource pool table after resource recovery at time t1.
[0088] When the actually used resources are much less than the applied resources, resource recycling will be triggered. When recycling resources, the recycled resources are not directly put back into the available resource pool of the machine, but temporarily placed in the alternative available resource pool of the machine. Only when the same task triggers resource recycling again next time, will the resources recycled last time be put back into the available resource pool, while the resources recycled this time will continue to be placed in the alternative available resource pool. This method avoids the situation where a task's memory usage is still increasing, but the resources are prematurely recycled (i.e., memory thrashing).
[0089] Resource recycling does not recover the difference between the applied resources and the actually used resources all at once, but in proportion and in batches. This method avoids the frequent recycling and new application of a task's resources, reduces the frequency of memory resource application, and thus improves the stability of the cluster system.
[0090] If the used resource U is much greater than the applied resource R, i.e., R - Bf < U, a new resource application will be triggered. At this time, the newly applied resource Nr = U + Bf - R;
[0091] In the formula, Nr is the newly applied resource, U is the used resource, R is the applied resource, and Bf is the reserved resource value for the task application;
[0092] Send a new application to apply for new resources. At this time, it will be divided into two cases according to the value of Rt:
[0093] a. If Nr < Rt, then this part of the newly added Nr resources can be directly allocated from Rt in Available_candidated. Note that allocating from Available_candidate here is to save the resources of Available, because Rt is the candidate resource of Available_candidate recycled by this task last time, but has not been put back into the Available available resources yet.
[0094] At this time, the resource application form Request of the container changes from Rm to Rm + Nr, New_request changes from 0 to Nr, and Return changes from Rt to Rt - Nr;
[0095] The Available_candidate of the resource pool changes from Rt to Rt - Nr, and Allocated changes from Rm - Rt to Rm - Rt + Nr, Figure 6 The change process of the values in the resource usage table and the machine resource pool table during the new resource application request stage is given.
[0096] b. If Nr > Rt, that is, the resources newly applied for are greater than the resources returned last time. At this time, the insufficient part needs to be allocated from the Available resource pool, and there will be two cases. Assume diff = Nr - Rt;
[0097] When the Available resource pool can meet this part of the difference, that is, T - Rm + Rt + diff > Nr, it means that Available meets the demand at this time. The changes in the resource usage table are that Request changes from Rm to Rm + Nr, Return changes from Rt to 0, and New_request changes from 0 to Nr. The Available in the resource pool changes from T - Rm to T - Rm + Rt - Nr, Available_candidate changes from Rt to 0, and Allocated changes from Rm - Rt to Rm - Rt + Nr; Figure 7 The change process of the values in the resource usage table and the machine resource pool table during the resource new application stage is given.
[0098] When the Available resource pool cannot meet this part of the difference, it will trigger the automatic stop of the current container, release the currently applied resources, and submit a request to apply for resources again with the latest U + Bf.
[0099] When the actual used resources are greater than the applied resources, it will trigger a new application for resources. At this time, the resources applied for by the task are not directly allocated from the available resources of the node, but first judge whether there has been a situation of resource recovery for this task before; if so, allocate from the alternative available resources first, and only when the alternative available resources are not enough, will it be allocated from the node available resources. In this way, when a task has large fluctuations in memory resource usage during operation, the resource utilization rate can be improved.
[0100] S50, the application ends, releases resources, and updates the resource usage table and the machine resource pool table.
[0101] At time t0, in the task resource usage table, the Request resource is Rm - Rt, the Used resource is 0, the Return resource in the previous round is Rt, and the new request resource in the previous round is Nr.
[0102] At this time, the value of Available in the resource pool is T - Rm, the value of Available_candidate is Rt, and the allocated resource Allocated is Rm - Rt.
[0103] After the resource is released at time t1, the container sends a request to update the released application resource Rm and release a part of the resource Return, that is, the Rt resource needs to be put back into the resource pool Available. At this time, the resource usage table Request becomes 0, Return becomes 0, and New_request becomes 0; while the resource in the resource pool changes from Available from T - Rm to T, Available_candidate from Rt to 0, and Allocated from Rm - Rt to 0. Figure 8 It shows the change process of the values of the resource usage table and the machine resource pool table when the application ends.
[0104] The explanations of the key technical terms in this solution are as follows:
[0105] container: The executable program submitted by the user to the cluster. Container is the basic unit of resources in YARN and contains resources such as CPU and memory.
[0106] Request: Represents the amount of memory resource requested.
[0107] New_request: Represents the amount of additional memory that needs to be requested when the actual memory used by the container is greater than the requested memory.
[0108] Used: Represents the actual amount of memory resources used.
[0109] Allocated represents the allocated resources of the machine.
[0110] The recycling ratio Ratio, abbreviated as RO, represents the ratio during each resource recycling. Resource recycling is carried out in stages. When the requested resources are greater than the used resources, the resource update recycling stage will intervene. However, not all the extra resources are recycled at once, but in proportion. This value can be set according to the actual resource usage pattern of specific cluster tasks.
[0111] Return: Represents the amount of memory that needs to be reduced when the actual memory used by the container is less than the requested memory.
[0112] Available: Represents the amount of available memory resources of the machine in the resource pool. The resource management component of YARN can know all the machines available in the current cluster and the available resources of each machine. Here, Available refers to the available resources of the machine allocated for the container to run.
[0113] Available_candidate represents the alternative available resources of the machine.
[0114] Allocated represents the allocated resources of the machine.
[0115] The reserved resource Buffer, abbreviated as Bf, represents a very small part of the resources set aside for task applications. Although the smaller the difference between the Request resource Rm and the used resource U, the better, we need to define a threshold in advance. When adjusted to a certain threshold, resource updates will not be triggered.
[0116] Resource application form: Request represents the memory resources that need to be applied for as specified by the user when submitting, and the initial value is Rm.
[0117] Resource usage form: Request represents the memory applied for by the task (container), with the value of Rm. Used represents the actual memory used by the task, which is U. Return represents the memory resources returned in the previous round, with the value of Rt (the initial value is 0, indicating that no memory was returned in the previous round), and is used when the applied memory is greater than the used memory and a part of the applied memory needs to be returned. New_request represents the newly required application, with the value of Nr (the initial value is 0, indicating no new application), and is used when the applied memory is less than the used memory and new application is required.
[0118] Resource pool: The resource pool represents the memory resource pool of a single node. Among them, Available represents the available memory of the current node, with the value of T (the initial value is the available resources configured when the machine Nodemanager process starts). Available_candidate represents the candidate available memory value of the current node, which is Ac (the initial value is 0). The reason for having candidate available memory is that when the applied resources are greater than the used resources, some resources will be recycled, but these resources will not be allocated to other tasks in the next resource recycling. Allocated represents the allocated memory, with the value of Al (the initial value is 0, indicating that no resources are allocated temporarily).
[0119] All the above formulas are dimensionless and take their numerical calculations. The formulas are obtained by collecting a large amount of data for software simulation to get a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.
[0120] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product.
[0121] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled professionals may use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.
[0122] In addition, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module.
[0123] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.
[0124] Finally: The above are only the preferred embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for dynamically updating memory resource application, characterized in that: The following steps are involved: Submit application requests, use the resource scheduler in YARN to process application requests, and start the task manager; Request to allocate required memory resources, the resource scheduler accepts the request and searches for machines that meet the resource requirements, and allocates the resource request through the resource pool; During the task execution, the actual memory resource usage of the application is obtained regularly, and the actual amount of memory resources used in the resource usage table is updated; Check the size relationship between the actual amount of memory resources used and the amount of memory resources required for application through a resource monitoring program, and update the resource request according to the size relationship between the actual amount of memory resources used and the amount of memory resources required for application; The application ends, resources are released, and the resource usage table and machine resource pool table are updated.
2. A method for dynamically updating memory resource application according to claim 1, characterized in that: The process for submitting an application request is as follows: When a client has an application to run, it will prepare relevant information about the application, including the memory resources required for the application to run.
3. A method for dynamically updating memory resource application according to claim 2, characterized in that: The process of using the resource scheduler in YARN to process application requests is as follows: After receiving the application request submitted by the client, the request is parsed and verified; The parsing and verification of the request includes checking the format of the request, various parameters, and the amount of memory resources declared by the client; And according to the scheduling algorithm of the resource scheduler and the resource usage of the system, plan how to allocate resources to the application.
4. A method for dynamically updating memory resource application according to claim 3, characterized in that: The process of starting the task manager is as follows: The resource scheduler starts the task manager; The task manager submits specific resource requests to YARN and interacts frequently with YARN's resource management module; Communicate specific information about the memory resources required by the application so that YARN can allocate resources to the application more accurately; After resource allocation is completed, the task manager monitors the resource usage of the application during operation and provides feedback to the resource scheduler.
5. A method for dynamically updating memory resource application according to claim 4, characterized in that: The request allocates the required memory resources, and the process of the resource scheduler accepting the request and finding a machine that meets the resource requirements is as follows: The client requests the required memory resources to be allocated, and the resource scheduler in YARN accepts the request and searches for a machine that meets the resource requirements; The machine that meets the resource requirements includes a machine whose available resources are greater than or equal to the requested resources; the initial available resources of the machine are the maximum available resources configured for the machine.
6. A method for dynamically updating memory resource application according to claim 5, characterized in that: The process of allocating resource requests through the resource pool is as follows: After resources are allocated, the resource pool status changes, and the available resources change from the original T to T-Rm, where T represents the initial available resources, Rm represents the allocated resources, and T-Rm represents the remaining available resources; The allocated resources change from 0 to Rm, and the requested allocated resources in the resource usage table change from 0 to Rm, where Rm indicates allocated resources.
7. A method for dynamically updating memory resource application according to claim 6, characterized in that: The steps of periodically obtaining the actual memory resource usage of the application and updating the actual memory resource usage in the resource usage table are as follows: The resource monitor is started periodically to collect the memory usage data of the application; The monitor obtains the actual currently used memory resources from each application; The monitor records the actual amount of memory resources used by the application in the resource usage table, and the resource usage table of each application is updated; And repeat this process at certain time intervals.
8. A method for dynamically updating memory resource application according to claim 7, characterized in that: The process of updating the resource request according to the relationship between the actual amount of memory resources used and the amount of memory resources required for the application is as follows: If the actual amount of memory resources used is much less than the amount of memory resources requested, the resource application update and recycling process is triggered; If the actual amount of memory resources used is much greater than the amount of memory resources required for the application, an additional application for resources is triggered. At this time, the newly applied resources Nr = U + Bf - R; Where Nr is the newly applied resource, U is the actual amount of memory resources used, R is the amount of memory resources required for the application, and Bf is the resource value reserved for the task application.
9. A method for dynamically updating memory resource application according to claim 8, characterized in that: If the actual amount of memory resources used is much less than the amount of memory resources required for the application, the resource application update and recycling process is triggered as follows: When the actual used resources are far less than the applied resources, resource recycling will be triggered. The resource recycling is done in batches according to the proportion. And temporarily place the recycled resources in the machine's alternative available resource pool; When the same task continues to trigger resource recycling next time, the resources recycled last time are put back into the available resource pool; The resources recycled this time will continue to be placed in the alternative available resource pool.
10. A method for dynamically updating memory resource application according to claim 9, characterized in that: If the actual amount of memory resources used is much greater than the amount of memory resources required for application, the process of triggering the additional application of resources is as follows: When the actual resources used are greater than the requested resources, a new resource request will be triggered to determine whether there has been a resource recycling situation before; And allocate from the alternative available resources. If the alternative available resources are insufficient, allocate from the node available resources.