Cache capacity control method, device, equipment and system

Abstract

The invention discloses a cache capacity control method, device, equipment and system, relates to the technical field of computers, and can improve the utilization rate of memory resources. The method can be applied to a cluster, the cluster comprises a scheduler management node and a plurality of computing nodes, the method is executed by the scheduler management node, and the method comprises the following steps: the scheduler management node receives a cache resource request sent by a first computing node, and allocates a cache resource to the first computing node according to the cache resource request and a resource allocation strategy; determining a target resource quantity allocated to a cache system of the first computing node from a computing memory of the first computing node to obtain a resource allocation result; and sending the resource allocation result to the first computing node. Wherein the resource allocation result is used for indicating that a target resource quantity in idle resources of the computing memory is divided into the cache system, and the resource allocation strategy comprises at least one of the following items: an occupation parameter of the computing memory, an occupation parameter of the cache system and a reserved resource quantity of the computing memory.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cache capacity control method, apparatus, device, and system. Background Art

[0002] During application execution, the central processing unit (CPU) runs much faster than the file system's input / output (I / O). When an application performs an I / O operation on the file system, it must pause and wait for the I / O operation to complete. This results in a longer end-to-end execution time for the application, which in turn means the application spends more time occupying computing resources, resulting in lower execution efficiency. To improve application execution efficiency and shorten end-to-end resource utilization, an I / O cache system is introduced to balance the performance differences between the CPU and the file system. By allocating a portion of the compute node's memory resources to serve as the I / O cache system, the speed difference between the CPU and the file system is mitigated, improving the overall execution efficiency of the application.

[0003] However, the capacity of the IO cache system is a fixed-capacity cache area configured from the memory resources of the computing node after careful consideration. This cache area cannot adapt to various tasks whose memory resource consumption changes all the time, resulting in low memory resource utilization. Summary of the Invention

[0004] The present application provides a cache capacity control method, apparatus, device and system, thereby improving the utilization of memory resources.

[0005] In a first aspect, a cache capacity control method is provided. The method can be applied to a cluster, where the cluster includes a scheduler management node and multiple computing nodes, and the method is executed by the scheduler management node.

[0006] The scheduler management node receives a cache resource request sent by the first computing node and, based on the cache resource request and a resource allocation policy, determines a target amount of resources to be allocated to the cache system of the first computing node from the computing memory of the first computing node, obtains a resource allocation result, and sends the resource allocation result to the first computing node. The resource allocation result indicates that the target amount of resources from the idle resources of the computing memory is to be allocated to the cache system, and the resource allocation policy includes at least one of the following: a computing memory occupancy parameter, a cache system occupancy parameter, and a reserved amount of computing memory resources.

[0007] Because, when the scheduler management node receives the cache resource request sent by the first computing node, it can determine the target resource amount allocated to the cache system of the first computing node from the computing memory of the first computing node based on the cache resource request and the occupancy parameters of the computing memory, the occupancy parameters of the cache system and the reserved resource amount of the computing memory, obtain the resource allocation result and send it to the first computing node; thereby, the first computing node allocates the target resource amount from the idle resources of the computing memory to the cache system based on the target resource amount according to the resource allocation result. Through the above method, when the cache capacity of the cache system in the first computing node is insufficient, a cache resource request can be sent to the scheduler management node to allocate the target resource amount from the computing memory of the first computing node to the cache system, thereby adjusting the resource allocation situation according to the cache system's demand for cache capacity. The utilization rate of memory resources by computing nodes can be improved.

[0008] In one possible design, the above-mentioned method of determining the target amount of resources to be allocated from the computing memory of the first computing node to the cache system of the first computing node based on the resource allocation strategy may specifically include: first, based on the resource allocation strategy, obtaining the occupancy parameters of the computing memory and the reserved amount of resources of the computing memory, and obtaining the occupancy parameters of the cache system and the amount of data to be stored, thereby determining the target amount of resources to be allocated from the computing memory to the cache system based on the occupancy parameters of the computing memory, the reserved amount of resources of the computing memory, the occupancy parameters of the cache system and the amount of data to be stored. Among them, the occupancy parameters of the cache system include: the total amount of resources of the cache system, the occupied amount of resources of the cache system and the idle amount of resources of the cache system; the occupancy parameters of the computing memory include: the total amount of resources of the computing memory, the occupied amount of resources of the computing memory and the idle amount of resources of the computing memory.

[0009] In this way, the scheduler management node can determine the target amount of resources that can be allocated to the cache system from the computing memory based on the total amount of resources, resource occupancy and amount of data to be stored in the cache system, as well as the total amount of resources, resource occupancy and reserved resources of the computing memory. The target amount of resources determined based on the above parameters can allocate the required resources to the cache system to the greatest extent without affecting the execution of computing jobs by the computing memory, thereby improving the utilization of memory resources by the computing nodes.

[0010] In one possible design, the method may also include: the scheduler management node receives a computing resource request sent by the first computing node for requesting computing resources of a first resource amount to execute a target job, and determines the size relationship between the free resource amount of the computing memory and the first resource amount; if the free resource amount of the computing memory is less than the first resource amount, a second computing node whose free resource amount of the computing memory is greater than or equal to the first resource amount can be determined from multiple computing nodes to allocate the target job to the second computing node for execution.

[0011] In this way, when a first computing node is assigned a target job that requires a first amount of computing resources, if the first computing node's free resources are less than the first amount, the target computing node can first determine whether there is a second computing node in the target computing cluster with a free computing memory resource equal to or greater than the first amount, and then assign the target job to the second computing node for execution. This eliminates the need to release the target amount of computing memory allocated to the cache system for reallocation to computing memory, thereby improving job execution efficiency.

[0012] In another possible design, the method may further include: if the idle resource amount of the computing memory included in each of the plurality of computing nodes is less than the first resource amount, the scheduler management node sending a cache resource release message to the first computing node, thereby instructing, via the cache resource release message, to allocate the second resource amount included in the cache system to the computing memory. The sum of the idle resource amount of the computing memory and the second resource amount is greater than or equal to the first resource amount, and the second resource amount is less than or equal to the target resource amount.

[0013] In this way, when it is determined that the amount of free resources of the first computing node is less than the first resource amount, and the amount of free resources of the computing memory included in each of the multiple computing nodes is less than the first resource amount, the scheduler management node can send a cache resource release message to the first computing node, so that the first computing node will release the resources occupied by the computing memory and reallocate them to the computing memory, thereby increasing the amount of free resources of the computing memory to execute the allocated target job, thereby improving the utilization of memory resources.

[0014] In a second aspect, a cache capacity control method is provided. The method can be applied to a cluster, where the cluster includes multiple computing nodes and a scheduler management node. The multiple computing nodes include a first computing node, and the method is executed by the first computing node.

[0015] First, the capacity of the cache system of the first computing node to be expanded is determined. The first computing node can then send a cache resource request to the scheduler management node. The cache resource request is used by the scheduler management node to determine the target amount of resources to be allocated to the cache system of the first computing node from the computing memory of the first computing node based on the resource allocation policy, and obtain a resource allocation result. The resource allocation result sent by the scheduler management node is then received, and the target amount of resources from the idle resources of the computing memory are allocated to the cache system. The resource allocation policy includes at least one of the following: computing memory occupancy parameters, cache system occupancy parameters, and computing memory reserved resource amount.

[0016] When the first computing node determines that the capacity of the cache system needs to be expanded, it can send a cache resource request to the scheduler management node, so that the scheduler management node determines the target amount of resources to be allocated to the cache system of the first computing node from the computing memory of the first computing node based on the occupancy parameters of the computing memory, the occupancy parameters of the cache system, and the reserved resource amount of the computing memory, and obtains a resource allocation result. Furthermore, the first computing node can allocate resources from the idle resources of the computing memory to the cache system based on the target resource amount according to the resource allocation result. Through the above method, when the cache capacity of the cache system in the first computing node is insufficient, a cache resource request can be sent to the scheduler management node to allocate the target amount of resources from the computing memory of the first computing node to the cache system, thereby adjusting the resource allocation situation according to the cache system's demand for cache capacity. The utilization rate of memory resources by computing nodes can be improved.

[0017] In one possible design, the above-mentioned determination of expanding the capacity of the cache system of the first computing node includes: the first computing node determining the cache system's occupancy parameters and the cache system's release parameters, and determining the target resource amount of the cache system to be expanded based on the cache system's occupancy parameters and the cache system's release parameters. The cache system's occupancy parameters include: the total amount of resources of the cache system, the amount of occupied resources of the cache system, the amount of idle resources of the cache system, and the average waiting time for resource occupancy of the cache system; and the cache system's release parameters include: the frequency of resource release of the cache system and the result of resource release of the cache system.

[0018] In this way, the first computing node can determine the target amount of resources to be expanded in the cache system based on the resource occupancy and resource release of the cache system, so that when sending a cache resource request to the scheduler management node, it can inform the scheduler management node of the amount of resources that the cache system needs to expand, thereby improving the efficiency of expanding resources in the cache system.

[0019] In one possible design, after allocating the target amount of resources in the idle resources of the computing memory to the cache system, the above method also includes: the first computing node receives a cache resource release message sent by the scheduler management node, which is used to indicate that the second amount of resources included in the cache system is allocated to the computing memory, and allocates the second amount of resources included in the cache system to the computing memory based on the cache resource release message.

[0020] In this way, if the cache system of the first computing node occupies computing memory resources, the occupied resources can be released and reallocated to computing memory based on the cache resource release message sent by the scheduler management node. This allows for flexible adjustment of the cache system and computing memory resource allocation, improving memory resource utilization.

[0021] In another possible design, after allocating the target amount of resources in the idle resources of the computing memory to the cache system, the above method also includes: the first computing node determines that the occupied amount of resources of the cache system is less than or equal to a preset threshold based on the occupancy parameter of the cache system and the release parameter of the cache system, and allocates the target amount of resources included in the cache system to the computing memory.

[0022] In this way, the first computing node can actively determine the resource occupancy of the cache system based on the occupancy parameters and release parameters of the cache system, and when the resource occupancy is small, actively allocate the amount of resources occupied by the cache system to the computing memory, thereby flexibly adjusting the amount of resources of the cache system and the computing memory and improving the utilization of memory resources.

[0023] According to a third aspect, a cache capacity control device is provided. The cache capacity control device is applied to a scheduler management node included in a cluster. The cluster also includes multiple computing nodes. The cache capacity control device includes: a first communication module and a first processing module.

[0024] The first communication module is configured to receive a cache resource request sent by the first computing node;

[0025] The above-mentioned first processing module is used to determine the target amount of resources allocated to the cache system of the first computing node from the computing memory of the first computing node according to the cache resource request and the resource allocation strategy, and obtain the resource allocation result. The resource allocation strategy includes at least one of the following: the occupancy parameters of the computing memory, the occupancy parameters of the cache system and the reserved resource amount of the computing memory.

[0026] The first communication module is further configured to send a resource allocation result to the first computing node, where the resource allocation result is used to indicate that a target amount of resources in the idle resources of the computing memory is allocated to the cache system.

[0027] In one possible design method, the above-mentioned first communication module is specifically used to obtain the occupancy parameters of the computing memory and the reserved resource amount of the computing memory based on the resource allocation strategy. The occupancy parameters of the computing memory include: the total resource amount of the computing memory, the occupied resource amount of the computing memory and the idle resource amount of the computing memory.

[0028] The first communication module is specifically used to obtain the occupancy parameters of the cache system and the amount of data to be stored. The occupancy parameters of the cache system include: the total resource amount of the cache system, the occupied resource amount of the cache system and the idle resource amount of the cache system.

[0029] The first processing module is specifically used to determine the target amount of resources allocated to the cache system from the computing memory based on the occupancy parameters of the computing memory, the reserved resources of the computing memory, the occupancy parameters of the cache system and the amount of data to be stored.

[0030] In another possible design, the first communication module is further configured to receive a computing resource request sent by the first computing node;

[0031] The first processing module is further configured to determine a size relationship between the idle resource amount of the computing memory and the first resource amount, and the computing resource request is configured to request computing resources of the first resource amount to execute a target job.

[0032] The first processing module is further configured to determine, from among the plurality of computing nodes, a second computing node whose idle memory resource amount is greater than or equal to the first resource amount if the idle memory resource amount is less than the first resource amount.

[0033] The first processing module is further configured to allocate the target job to the second computing node for execution.

[0034] In another possible design, the above-mentioned first communication module is also used to send a cache resource release message to the first computing node if the amount of free resources of the computing memory included in each of the multiple computing nodes is less than the first resource amount. The cache resource release message is used to indicate: the second resource amount included in the cache system is divided into the computing memory, the sum of the free resource amount of the computing memory and the second resource amount is greater than or equal to the first resource amount, and the second resource amount is less than or equal to the target resource amount.

[0035] In a fourth aspect, a cache capacity control device is provided, which is applied to a first computing node among multiple computing nodes included in a cluster. The cluster also includes a scheduler management node. The cache capacity control device includes: a second processing module and a second communication module.

[0036] The second processing module is configured to determine the capacity of the cache system of the expanded first computing node;

[0037] The above-mentioned second communication module is used to send a cache resource request to the scheduler management node. The cache resource request is used by the scheduler management node to determine the target amount of resources to be allocated to the cache system of the first computing node from the computing memory of the first computing node based on the resource allocation strategy, and obtain the resource allocation result. The resource allocation strategy includes at least one of the following: the occupancy parameters of the computing memory, the occupancy parameters of the cache system and the reserved resource amount of the computing memory.

[0038] The second communication module is further configured to receive a resource allocation result sent by the scheduler management node.

[0039] The second processing module is further configured to allocate a target amount of resources from the idle resources of the computing memory to the cache system.

[0040] In one possible design method, the above-mentioned second processing module is specifically used to determine the occupancy parameters of the cache system and the release parameters of the cache system. The occupancy parameters of the cache system include: the total resource amount of the cache system, the occupied resource amount of the cache system, the idle resource amount of the cache system and the average waiting time of resource occupancy of the cache system. The release parameters of the cache system include: the resource release frequency of the cache system and the resource release result of the cache system.

[0041] The second processing module is specifically configured to determine a target amount of resources to be expanded for the cache system based on an occupancy parameter of the cache system and a release parameter of the cache system.

[0042] In a possible design, the second communication module is further configured to receive a cache resource release message sent by the scheduler management node, where the cache resource release message is configured to indicate that the second resource amount included in the cache system is allocated to the computing memory.

[0043] The second processing module is further configured to allocate the second resource amount included in the cache system to the computing memory.

[0044] In another possible design, the second processing module is further configured to determine whether the amount of occupied resources of the cache system is less than or equal to a preset threshold based on an occupancy parameter of the cache system and a release parameter of the cache system.

[0045] The second processing module is further configured to allocate the target resource amount included in the cache system to the computing memory.

[0046] In a fifth aspect, an electronic device is provided, which is a scheduler management node or a first computing node, and the electronic device includes a memory and a processor, and the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method described in the first aspect and any possible design method thereof, or the method described in the second aspect and any possible design method thereof.

[0047] In the sixth aspect, a cache capacity control system is provided, which includes multiple computing nodes, and the multiple computing nodes include a scheduler management node and a first computing node. The cache capacity control system is used to execute the method described in the first aspect and any possible design method thereof, or the method described in the second aspect and any possible design method thereof.

[0048] In the seventh aspect, a computer storage medium is provided, which includes computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the method described in the first aspect and any possible design method thereof, or the method described in the second aspect and any possible design method thereof.

[0049] In an eighth aspect, a computer program product is provided, which, when run on a computer, enables the computer to execute the method described in the first aspect and any possible design thereof, or the method described in the second aspect and any possible design thereof.

[0050] It can be understood that the beneficial effects that can be achieved by the method described in the second aspect and any possible design thereof provided above, the cache capacity control device described in the third aspect and any possible design thereof, the cache capacity control device described in the fourth aspect and any possible design thereof, the electronic device described in the fifth aspect, the cache capacity control system described in the sixth aspect, the computer storage medium described in the seventh aspect, and the computer program product described in the eighth aspect can be referred to the beneficial effects in the first aspect and any possible design thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the composition of a cache capacity control system architecture provided by this application;

[0052] Figure 2 A schematic diagram of a cache capacity control method provided in this application;

[0053] Figure 3 A schematic diagram of another cache capacity control system architecture provided by this application;

[0054] Figure 4 A schematic diagram of the composition of a cache capacity control device provided by this application;

[0055] Figure 5 A schematic diagram of another cache capacity control device provided by this application;

[0056] Figure 6 A schematic diagram of the structural composition of an electronic device provided in this application. DETAILED DESCRIPTION

[0057] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] In this application, the character " / " generally indicates that the preceding and following objects are in an "or" relationship. For example, A / B can be understood as A or B.

[0059] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0060] Furthermore, the terms "including," "having," and any variations thereof, as used in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include other steps or modules not listed, or may optionally include other steps or modules inherent to the process, method, product, or apparatus.

[0061] Additionally, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0062] Currently, file caching technology utilizes memory or fast storage devices as an intermediary layer (i.e., an IO cache system) to balance the performance gap between computing and storage. IO cache systems typically operate between applications and file systems. Because the CPU runs much faster than the file system's IO speed during application execution, applications must pause and wait for IO operations to complete, resulting in longer end-to-end execution times. In high-performance computing clusters, this increased end-to-end time also means that applications spend more time using computing resources. To accelerate application execution efficiency and shorten end-to-end resource usage, high-speed storage devices are often used as IO cache systems to mitigate the speed gap between the CPU and file system, thereby improving overall application execution efficiency. Furthermore, in large-scale high-performance computing clusters, multi-tiered storage systems are often employed to balance efficiency and capacity expansion. Furthermore, in large distributed computing clusters, the scheduler is an essential component. It collects and coordinates the cluster's global computing resources, receives resource requests from various computing jobs, allocates resources appropriately based on scheduling policies and algorithms, and schedules computing tasks to the allocated computing resources, ultimately managing the lifecycle of computing tasks.

[0063] Taking memory caching as an example, because memory caching requires additional memory to support caching, it inevitably competes with compute jobs for memory resources. As the memory used for caching increases, cache performance improves, but the amount of memory available for computing decreases. On the other hand, if the cache memory is too small, cache performance is poor, and the benefits of IO operations decrease. Current memory-based IO caching systems have a fixed cache capacity configured after careful consideration. When configuring available memory for compute nodes, the scheduler preemptively excludes memory used for IO caching. This avoids competition between compute jobs and IO caching for memory resources during runtime.

[0064] In large computing clusters, due to the peaks and valleys in job scheduling, many jobs use CPU and memory resources unevenly over time. Some jobs running on some compute nodes may consume more CPU and less memory, or vice versa. This can cause compute nodes to have excessively high CPU utilization and low memory utilization during certain periods of time, or vice versa. For example, electronic design automation (EDA) clusters face this problem, resulting in overall cluster memory utilization of only around 30% to 40% from a monitoring and maintenance perspective. However, to cope with resource consumption during peak computing periods, higher memory resources must be configured.

[0065] Currently, using IO cache systems to allocate a certain amount of memory for caching can achieve a multiple-fold increase in system IO performance. This approach trades memory resources for computing efficiency. The IO cache system's IO performance acceleration is often positively correlated with the amount of memory available for caching. However, on a compute node, the IO cache system is intended to assist the computational process and should not compromise the efficiency of computational tasks by occupying excessive memory resources. Therefore, typical IO cache systems, after careful analysis and trade-offs, configure a fixed amount of memory as the IO cache system's cache area to minimize or control the impact of the IO cache system on the efficiency of normal computing operations. However, in practice, the memory consumption required by computing operations varies constantly across time periods and business types. When computing operations consume less memory, the cache cannot utilize the excess memory, resulting in wasted memory resources on the server. Over a longer period of time, the server's average memory utilization efficiency is relatively low.

[0066] To this end, embodiments of the present application provide a cache capacity control method. By coordinating the cache system with the scheduler, the cache system dynamically adds excess memory resources on the compute nodes to expand the cache system's memory space and improve cache performance. Furthermore, when compute node memory resources are limited, the cache system dynamically releases memory, preventing the cache system from preempting computing resources.

[0067] This method can be applied to the process of controlling the cache capacity of the cache system of the computing node. In the present application, when the scheduler management node receives a cache resource request sent by the first computing node, it can determine the target resource amount to be allocated for the cache system of the first computing node from the computing memory of the first computing node based on the occupancy parameters of the computing memory, the occupancy parameters of the cache system and the reserved resource amount of the computing memory according to the cache resource request, obtain the resource allocation result and send it to the first computing node; thereby, the first computing node allocates the target resource amount in the free resources of the computing memory to the cache system based on the target resource amount according to the resource allocation result. Through the above method, when the cache capacity of the cache system in the first computing node is insufficient, a cache resource request can be sent to the scheduler management node to allocate the target resource amount from the computing memory of the first computing node to the cache system, thereby adjusting the resource allocation situation according to the cache system's demand for cache capacity. The utilization rate of memory resources by computing nodes can be improved.

[0068] refer to Figure 1 The cache capacity control method provided in the embodiment of the present application can be applied to the implementation environment of a computing cluster composed of multiple computing nodes (i.e., a cache capacity control system). Figure 1As shown, the implementation environment may include a scheduler management node 101, a first computing node 102 and other computing nodes 103. Among them, the scheduler included in the scheduler management node 101 includes: a scheduling policy module and a resource acquisition module. The scheduling policy module is specifically a cache memory scheduling policy, which is used to adjust the cache capacity of the cache system included in the first computing node 102 and / or other computing nodes 103; the resource acquisition module is used to obtain the actual resource usage information of the first computing node 102 and / or other computing nodes 103 (for example, the occupancy of the computing memory, or the memory occupancy of the cache system). The first computing node 102 includes a cache system, and the cache system includes a capacity dynamic policy management module and a cache space management module. Among them, the capacity dynamic policy management module may specifically include: a capacity expansion policy management module and a capacity contraction policy management module; the cache space management module may specifically include: a cache memory add / delete interface, a cache space allocation extension module, a cache memory status query interface, a contraction policy support module, a cache data persistence module and a contraction space recovery module. Each of the other computing nodes 103 is the same as the first computing node 102 and is a computing node including a cache system. For details, please refer to the description of the first computing node 102 and will not be repeated here.

[0069] Optionally, the scheduling policy module increases the ability to allocate and reclaim memory resources of the cache system, and supports giving priority to avoidance strategies (i.e., allocating jobs to computing nodes whose other resources meet the requirements) when making scheduling decisions, so that the memory resources occupied by the cache system cannot be actively reclaimed.

[0070] Optionally, the capacity expansion policy management module is specifically responsible for determining whether cache space expansion is necessary based on the cache space usage of the compute node's cache system. This determination is based on the cache system's occupancy parameters and cache system release parameters. These metrics reflect the current level of urgency in the cache system's memory usage. Based on this information, the cache system's required memory size can be calculated and packaged into a cache resource request, which is then sent to the scheduler management node.

[0071] Optionally, there are two ways to trigger the capacity shrinkage policy management module, one is triggered by the scheduler management node, and the other is triggered by the cache system. Specifically, when the scheduler management node is triggered, when the scheduler management node cannot find suitable computing resources to allocate computing tasks, it can calculate whether the resource allocation requirements can be met after excluding the resources occupied by the cache system; if satisfied, a cache resource release message is sent to the cache system on the computing node, so that the capacity shrinkage policy management module starts to release the memory resources of the cache system after receiving the message sent by the scheduler management node. When the cache system is triggered, the cache system can release cache resources when it determines that its own consumption of cache space has decreased and the cache load has been reduced. The specific judgment is based on the occupancy parameters of the cache system and the release parameters of the cache system. The expansion and contraction standards can be determined by the difference in the parameter values. And after the cache system releases the memory space, the result is fed back to the scheduler management node.

[0072] Optionally, the cache space management module included in the cache system is an inherent module of the cache system, which includes an extended cache memory add / delete interface, a cache space allocation extension module, a cache memory status query interface, and a shrinking policy support module. Specifically, the cache memory add / delete interface is used for the capacity dynamic policy management module to call for policy driving; the cache memory status query interface is used for the capacity dynamic policy management module to call for policy judgment; the cache space allocation extension module is an extension of the cache space allocation capability, supporting the allocation of cache space to requests from dynamically changing cache space; the shrinking policy support module is used to give priority to the memory space to be released when the cache system releases memory resources, to ensure that the resource release operation is completed as soon as possible.

[0073] For example, the computing nodes in the embodiments of the present application may be tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), augmented reality (AR) and virtual reality (VR) devices, and the embodiments of the present application do not impose any special restrictions on the specific form of the devices.

[0074] The execution subject of the cache capacity control method provided in the present application can be the central processing unit (CPU) of the computing node, or the control module in the computing node for implementing cache capacity control, or the application system in the computing node for implementing cache capacity control.

[0075] The computing nodes in the embodiments of the present application are used to execute jobs and store data information. The computing nodes can be servers, desktops, laptops, mobile phones, cloud devices, or virtual machines. They can provide access to service logic for use by client application programs (systems). Computing nodes can provide applications with a simple and manageable access mechanism to system resources. They also provide services such as Hypertext Transfer Protocol (HTTP) implementation and database connection management.

[0076] The technical solutions provided in the embodiments of this application can be applied to the above-mentioned implementation environment. The implementation environment described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application and does not constitute a limitation on the technical solutions provided in the embodiments of this application. It is known to those skilled in the art that as the implementation environment evolves, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0077] The methods in the following embodiments can all be implemented in a computing node with the above implementation environment. In the following embodiments, the execution subject of the cache capacity control method provided by the present application is a scheduler management node and / or a first computing node as an example to illustrate the methods of the embodiments of the present application.

[0078] The embodiment of the present application provides a cache capacity control method, which is applied to a cluster, wherein the cluster includes a scheduler management node and multiple computing nodes, wherein the multiple computing nodes include a first computing node, and the method is executed by the scheduler management node or the first computing node, such as Figure 2 As shown, the cache capacity control method may include S201-S204.

[0079] S201: Determine to expand the capacity of the cache system of the first computing node, and the first computing node sends a cache resource request to the scheduler management node.

[0080] Among them, the cache resource request is used by the scheduler management node to determine the target amount of resources to be allocated to the cache system of the first computing node from the computing memory of the first computing node based on the resource allocation strategy, and obtain the resource allocation result. The resource allocation strategy includes at least one of the following: the occupancy parameters of the computing memory, the occupancy parameters of the cache system and the reserved resource amount of the computing memory. The reserved resource amount of the computing memory is used by the first computing node to perform other job tasks.

[0081] Optionally, the scheduler management node includes any of the following schedulers: Donau, LSF (Load Sharing Facility), or Slurm (Simple Linux Utility for Resource Management). LSF is a distributed cluster management system software responsible for managing computing resources and scheduling processing jobs; Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system suitable for large and small Linux clusters.

[0082] The embodiment of the present application takes the Danube scheduler as an example to control the cache capacity of the cache system, which can be applied to a high-performance computing cluster, and the cache system can be an IO cache system or other forms of cache systems. The high-performance cluster uses the Danube scheduler to schedule high-performance computing (HPC) jobs, and introduces a cache system to improve the IO performance of HPC jobs. During use, through the cooperation of the cache system and the Danube scheduler, the memory resources of the computing nodes in the computing cluster during the time period with surplus memory resources are fully utilized to perform IO acceleration. On the one hand, IO acceleration improves the end-to-end execution time of HPC jobs, which is conducive to more efficient and practical computing resources; on the other hand, by utilizing surplus memory resources, the overall utilization efficiency of the computing cluster's memory resources can also be improved.

[0083] like Figure 3 As shown, the scheduler management node includes a scheduler, which includes a scheduling policy module; the first compute node includes an IO cache system, a scheduler agent, and a scheduler command-line interface (CLI); each of the multiple compute nodes included in the target compute cluster includes an IO cache system, a scheduler agent, and a scheduler command-line interface (CLI), providing IO operation cache acceleration services for application processes running on the compute node. The scheduler agent and scheduler command line work in conjunction with the IO cache system to achieve elastic scaling of the IO cache system's cache capacity.

[0084] It should be noted that the cache system will be configured with a default cache capacity to cache and accelerate IO operations. When it is determined that the capacity of the cache system of the first computing node needs to be expanded, a cache resource request can be sent to the scheduler command line through the cache system, and the cache resource request can be forwarded to the scheduler in the scheduler management node through the scheduler command line.

[0085] Optionally, the first computing node first needs to determine the occupancy parameters and release parameters of the cache system, and then, based on the occupancy parameters and release parameters of the cache system, the first computing node determines the target resource amount to be expanded of the cache system.

[0086] Among them, the cache system occupancy parameters include: the total resource amount of the cache system, the occupied resource amount of the cache system, the idle resource amount of the cache system and the average waiting time of resource occupancy of the cache system; the cache system release parameters include: the resource release frequency of the cache system and the resource release result of the cache system.

[0087] It should be noted that the average waiting time for resource occupation of the cache system can be understood as: when a cache task needs to occupy storage resources, the average time the cache system allocates storage resources to the cache task. The resource release frequency of the cache system can be understood as: when the cache capacity of the cache system is occupied, each time a cache task needs to occupy storage resources, the occupied memory resources need to be released (cleared) before the storage resources are allocated to the cache task, that is, the resource release frequency of the cache system is the frequency of triggering the release (clearance) of the occupied memory resources. The resource release result of the cache system can be understood as: when the occupied memory resources need to be released (cleared), whether the data stored in the memory of the cache system can be released (cleared). The resource release result of the cache system can include: successful resource release and failed resource release.

[0088] S202. The scheduler management node receives the cache resource request sent by the first computing node, determines the target amount of resources allocated to the cache system of the first computing node from the computing memory of the first computing node according to the cache resource request and the resource allocation strategy, and obtains the resource allocation result.

[0089] The resource allocation strategy includes at least one of the following: an occupancy parameter of the computing memory, an occupancy parameter of the cache system, and a reserved resource amount of the computing memory, where the reserved resource amount of the computing memory is used by the first computing node to execute other job tasks.

[0090] In one possible implementation, the first computing node needs to obtain computing memory occupancy parameters and the amount of computing memory reserved resources based on the resource allocation policy, as well as the cache system occupancy parameters and the amount of data to be stored. Then, based on the computing memory occupancy parameters, the amount of computing memory reserved resources, the cache system occupancy parameters, and the amount of data to be stored, the first computing node determines a target amount of resources to be allocated from the computing memory to the cache system.

[0091] The memory usage parameters include: the total memory resources, the memory usage, and the free memory resources; the cache system usage parameters include: the total memory resources, the memory usage, and the free memory resources;

[0092] It should be noted that each job task has a corresponding execution priority. When a high-priority job task needs to reserve resources, the corresponding amount of reserved resources needs to be reserved in the computing memory for the high-priority job task.

[0093] Combine Figure 3 After the scheduler in the scheduler management node receives the cache resource request, it can determine the target resource amount allocated to the IO cache system of the first computing node from the computing memory of the first computing node according to the resource allocation policy.

[0094] For example, assuming that the computing memory has a total of 100G of resources, 50G of which is occupied, and 20G of resources needs to be reserved for high-priority job tasks, the cache system has a total of 20G of resources, and 15G of which is occupied. If the cache resource request sent by the first computing node is used to request 10G of resources, since there are 30G of resources left in the computing memory, 10G of resources are allowed to be allocated to the cache system; if the computing memory is occupied by 50G of resources, 75G of resources, and 20G of resources needs to be reserved for high-priority job tasks, only 5G of resources are left. When the cache resource request sent by the first computing node is used to request 10G of resources, only 5G of resources can be allocated to the cache system.

[0095] S203: The scheduler management node sends the resource allocation result to the first computing node.

[0096] The resource allocation result is used to indicate: allocating a target amount of resources in the idle resources of the computing memory to the cache system.

[0097] It can be understood that the resource allocation result is specifically used to instruct the first computing node to allocate resources in the idle resources of the computing memory to the cache system based on the target resource amount.

[0098] Combine Figure 3 The scheduler in the scheduler management node sends the obtained resource allocation result to the scheduler agent in the first computing node, so that the resource allocation result is sent to the IO cache system through the scheduler agent and the scheduling command is executed to allocate memory resources.

[0099] S204: The first computing node receives the resource allocation result sent by the scheduler management node, and allocates the target amount of resources in the idle resources of the computing memory to the cache system.

[0100] That is, the first computing node allocates resources from the idle resources of the computing memory to the cache system based on the target resource amount indicated by the resource allocation result.

[0101] Optionally, when the first computing node receives an assigned computing job, the first computing node needs to allocate computing resources for the computing job. When the free computing resources of the computing memory of the first computing node are insufficient, the first computing node can send a computing resource request to the scheduler management node so that the scheduler management node triggers the allocation of resources allocated to the cache system to the computing memory.

[0102] In one possible implementation, the scheduler management node can receive a computing resource request sent by a first computing node and determine the size relationship between the amount of free resources in the computing memory and the first resource amount; then, when it is determined that the amount of free resources in the computing memory is less than the first resource amount, a second computing node whose amount of free resources in the computing memory is greater than or equal to the first resource amount can be determined from multiple computing nodes, and the target job can be assigned to the second computing node for execution.

[0103] The computing resource request is used to request computing resources of a first amount of resources to execute a target job.

[0104] It is understandable that the scheduler management node needs to first determine whether it can determine the second computing node with free resources of computing memory from other computing nodes to meet the needs of the target job, and then allocate the target job to the second computing node for execution.

[0105] One possible implementation method is to send a cache resource release message to the first computing node when it is determined that the amount of free resources of the computing memory included in each computing node among multiple computing nodes is less than the first resource amount. The cache resource release message is used to indicate: the second resource amount included in the cache system is allocated to the computing memory.

[0106] The sum of the idle resource amount of the computing memory and the second resource amount is greater than or equal to the first resource amount, and the second resource amount is less than or equal to the target resource amount.

[0107] Specifically, the cache resource release message may instruct the first computing node to allocate the second resource amount of resources included in the cache system to the computing memory.

[0108] It can be understood that when the amount of free resources in the computing memory of each computing node in other computing nodes cannot meet the amount of resources required by the target job, the target job needs to be retained in the first computing node for execution, and the resources occupied by the cache system need to be reallocated to the computing memory.

[0109] Optionally, the scheduler management node may proactively trigger the reallocation of resources occupied by the cache system to the computing memory based on the occupancy of the computing memory of the first computing node.

[0110] In one possible implementation, when the scheduler management node determines that the resources occupied by the cache system need to be reallocated to the computing memory, it can send a cache resource release message to the first computing node, so that when the first computing node receives the cache resource release message sent by the scheduler management node, the second amount of resources included in the cache system can be allocated to the computing memory.

[0111] The cache resource release message is used to indicate that the second resource amount included in the cache system is allocated to the computing memory.

[0112] Specifically, the cache resource release message may be used to instruct the first computing node to allocate the second resource amount of resources included in the cache system to the computing memory.

[0113] Optionally, the first computing node may also actively determine whether it needs to continue to occupy the resources of the computing memory based on the occupancy parameters of the cache system and the release parameters of the cache system, so that based on the occupancy parameters of the cache system and the release parameters of the cache system, when it is determined that the amount of occupied resources of the cache system is less than or equal to a preset threshold, the target amount of resources included in the cache system may be allocated to the computing memory.

[0114] Optional, combined Figure 3 When the scheduler management node actively triggers the reallocation of resources occupied by the IO cache system to the computing memory based on the occupancy of the computing memory of the first computing node, the scheduler in the scheduler management node can send a cache resource release message to the scheduler agent of the first computing node, so that the cache resource release message can be sent to the IO cache system through the scheduler agent; or, when the first computing node actively determines that the resources occupying the computing memory need to be reallocated to the computing memory, a cache resource release message can be sent to the scheduler in the scheduler management node through the scheduler agent to allocate the resources of the second resource amount included in the IO cache system to the computing memory.

[0115] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the scheduler management node and the first computing node. It can be understood that, in order to realize the above functions, the scheduler management node and the first computing node include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the steps of a cache capacity control method of each example described in the embodiment disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or electronic device software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0116] In the embodiment of the present application, the scheduler management node and the first computing node can be divided into functional modules or functional units according to the above method example. For example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0117] Please refer to Figure 4 , which shows a schematic diagram of a cache capacity control device provided by an embodiment of the present application, the cache capacity control device is applied to a scheduler management node included in a cluster, and the cluster also includes multiple computing nodes. Figure 4 As shown, the cache capacity control device 400 may include: a first communication module 401 and a first processing module 402 .

[0118] The first communication module 401 is used by the cache capacity control device 400 to receive a cache resource request sent by the first computing node. For example, the first communication module 401 is used to support the cache capacity control device 400 in executing S202 in the above method embodiment and / or other processes of the technology described herein.

[0119] First processing module 402 is configured for cache capacity control device 400 to determine, based on the cache resource request and resource allocation policy, a target resource amount to be allocated to the cache system of the first computing node from the computing memory of the first computing node, and obtain a resource allocation result. The resource allocation policy includes at least one of the following: a computing memory occupancy parameter, a cache system occupancy parameter, and a reserved computing memory resource amount. For example, first processing module 402 is configured to support cache capacity control device 400 in executing S202 of the aforementioned method embodiment and / or other processes used in the techniques described herein.

[0120] First communication module 401 is further configured to enable cache capacity control device 400 to transmit a resource allocation result to the first computing node. The resource allocation result indicates that a target amount of idle resources in the computing memory should be allocated to the cache system. For example, first communication module 401 is configured to support cache capacity control device 400 in executing S203 of the aforementioned method embodiment and / or other processes of the technology described herein.

[0121] In one possible design method, the above-mentioned first communication module 401 is specifically used to obtain the occupancy parameters of the computing memory and the reserved resource amount of the computing memory based on the resource allocation strategy. The occupancy parameters of the computing memory include: the total resource amount of the computing memory, the occupied resource amount of the computing memory and the idle resource amount of the computing memory.

[0122] The first communication module 401 is specifically used to obtain the occupancy parameters of the cache system and the amount of data to be stored. The occupancy parameters of the cache system include: the total resource amount of the cache system, the occupied resource amount of the cache system and the idle resource amount of the cache system.

[0123] The first processing module 402 is specifically used to determine the target amount of resources allocated to the cache system from the computing memory based on the computing memory occupancy parameter, the computing memory reserved resource amount, the cache system occupancy parameter and the amount of data to be stored.

[0124] In another possible design, the first communication module 401 is further configured to receive a computing resource request sent by the first computing node;

[0125] The first processing module 402 is further configured to determine a size relationship between the idle resource amount of the computing memory and the first resource amount, and the computing resource request is configured to request computing resources of the first resource amount to execute a target job.

[0126] The first processing module 402 is further configured to determine, from among the plurality of computing nodes, a second computing node whose free memory resource amount is greater than or equal to the first resource amount if the free memory resource amount is less than the first resource amount.

[0127] The first processing module 402 is further configured to allocate the target job to the second computing node for execution.

[0128] In another possible design, the above-mentioned first communication module is also used to send a cache resource release message to the first computing node if the amount of free resources of the computing memory included in each of the multiple computing nodes is less than the first resource amount. The cache resource release message is used to indicate: the second resource amount included in the cache system is divided into the computing memory, the sum of the free resource amount of the computing memory and the second resource amount is greater than or equal to the first resource amount, and the second resource amount is less than or equal to the target resource amount.

[0129] Please refer to Figure 5 , which shows a schematic diagram of a cache capacity control device provided by an embodiment of the present application, the cache capacity control device is applied to the first computing node among multiple computing nodes included in the cluster, and the cluster also includes a scheduler management node. Figure 5 As shown, the cache capacity control device 500 may include: a second processing module 501 and a second communication module 502 .

[0130] The second processing module 501 is used by the cache capacity control device 500 to determine the capacity of the cache system of the first computing node. For example, the second processing module 501 is used to support the cache capacity control device 500 in performing S201 in the above-mentioned method embodiment and / or other processes of the technology described herein.

[0131] The second communication module 502 is used by the cache capacity control device 500 to execute: sending a cache resource request to the scheduler management node. The cache resource request is used by the scheduler management node to determine, based on a resource allocation policy, a target amount of resources to be allocated from the computing memory of the first computing node to the cache system of the first computing node, and obtain a resource allocation result. The resource allocation policy includes at least one of the following: a computing memory occupancy parameter, a cache system occupancy parameter, and a reserved amount of computing memory resources. For example, the second communication module 502 is used to support the cache capacity control device 500 in executing S202 of the aforementioned method embodiment and / or other processes of the technology described herein.

[0132] The second communication module 502 is further configured for the cache capacity control device 500 to receive the resource allocation result sent by the scheduler management node. For example, the second communication module 502 is configured to support the cache capacity control device 500 in executing S204 in the above method embodiment and / or other processes of the technology described herein.

[0133] The second processing module 501 is further configured for the cache capacity control device 500 to allocate a target amount of idle resources in the computing memory to the cache system. For example, the second processing module 501 is configured to support the cache capacity control device 500 in executing S204 in the aforementioned method embodiment and / or other processes in the technology described herein.

[0134] In one possible design method, the above-mentioned second processing module 501 is specifically used to determine the occupancy parameters of the cache system and the release parameters of the cache system. The occupancy parameters of the cache system include: the total resource amount of the cache system, the occupied resource amount of the cache system, the idle resource amount of the cache system and the average waiting time of resource occupancy of the cache system. The release parameters of the cache system include: the resource release frequency of the cache system and the resource release result of the cache system.

[0135] The second processing module 501 is specifically configured to determine a target amount of resources to be expanded for the cache system based on an occupancy parameter and a release parameter of the cache system.

[0136] In a possible design, the second communication module 502 is further configured to receive a cache resource release message sent by the scheduler management node, where the cache resource release message is configured to indicate that the second resource amount included in the cache system is allocated to the computing memory.

[0137] The second processing module 501 is further configured to allocate the second resource amount included in the cache system to the computing memory.

[0138] In another possible design, the second processing module 501 is further configured to determine whether the amount of occupied resources of the cache system is less than or equal to a preset threshold based on an occupancy parameter of the cache system and a release parameter of the cache system.

[0139] The second processing module 501 is further configured to allocate the target resource amount included in the cache system to the computing memory.

[0140] Other embodiments of the present application provide an electronic device. The electronic device is a scheduler management node or a first computing node. The electronic device may include a memory and one or more processors. The memory and the processor are coupled. The memory is configured to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device may perform the functions or steps performed by the client in the above-described method embodiments.

[0141] The embodiment of the present application further provides an electronic device 600, such as Figure 6As shown, the electronic device 600 includes at least one processor 601 and at least one interface circuit 602. The processor 601 and the interface circuit 602 can be interconnected via lines. For example, the interface circuit 602 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 602 can be used to send signals to other devices (such as the processor 601). Exemplarily, the interface circuit 602 can read instructions stored in the memory and send the instructions to the processor 601. When the instructions are executed by the processor 601, the electronic device can execute the various steps in the above embodiments. Of course, the electronic device can also include other discrete components, which is not specifically limited in the embodiments of the present application.

[0142] An embodiment of the present application further provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes the various functions or steps executed by the mobile phone in the above-mentioned method embodiment.

[0143] The embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the mobile phone in the above method embodiment.

[0144] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0149] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A cache capacity control method, characterized in that: Applied to a cluster, the cluster includes a scheduler management node and multiple computing nodes, the method is executed by the scheduler management node, and the method includes: receiving a cache resource request sent by the first computing node; determining, based on the cache resource request and the resource allocation policy, a target resource amount to be allocated to the cache system of the first computing node from the computing memory of the first computing node, and obtaining a resource allocation result, wherein the resource allocation policy includes at least one of the following: an occupancy parameter of the computing memory, an occupancy parameter of the cache system, and a reserved resource amount of the computing memory; The resource allocation result is sent to the first computing node, where the resource allocation result is used to indicate that the target amount of resources in the idle resources of the computing memory is allocated to the cache system.

2. The method according to claim 1, characterized in that The determining, from the computing memory of the first computing node, a target amount of resources to be allocated to the cache system of the first computing node according to the cache resource request and the resource allocation policy, includes: Based on the resource allocation strategy, obtaining the occupancy parameters of the computing memory and the reserved resource amount of the computing memory, wherein the occupancy parameters of the computing memory include: the total resource amount of the computing memory, the occupied resource amount of the computing memory, and the idle resource amount of the computing memory; Obtaining occupancy parameters of the cache system and the amount of data to be stored, wherein the occupancy parameters of the cache system include: the total amount of resources of the cache system, the amount of occupied resources of the cache system, and the amount of idle resources of the cache system; The target amount of resources allocated from the computing memory to the cache system is determined based on the occupancy parameter of the computing memory, the reserved resource amount of the computing memory, the occupancy parameter of the cache system and the amount of data to be stored.

3. The method according to claim 1 or 2, characterized in that The method further comprises: receiving a computing resource request sent by the first computing node, and determining a size relationship between the idle resource amount of the computing memory and the first resource amount, wherein the computing resource request is used to request computing resources of the first resource amount to execute a target job; If the amount of free resources of the computing memory is less than the first amount of resources, determining a second computing node from the plurality of computing nodes whose amount of free resources of the computing memory is greater than or equal to the first amount of resources; Allocate the target job to the second computing node for execution.

4. The method according to claim 3, characterized in that The method further comprises: If the amount of free resources of the computing memory included in each of the multiple computing nodes is less than the first resource amount, a cache resource release message is sent to the first computing node, and the cache resource release message is used to indicate: the second resource amount included in the cache system is divided into the computing memory, the sum of the free resource amount of the computing memory and the second resource amount is greater than or equal to the first resource amount, and the second resource amount is less than or equal to the target resource amount.

5. A cache capacity control method, characterized in that: Applied to a cluster, the cluster includes multiple computing nodes and a scheduler management node, the multiple computing nodes include a first computing node, the method is executed by the first computing node, and the method includes: Determine to expand the capacity of the cache system of the first computing node, and send a cache resource request to the scheduler management node, wherein the cache resource request is used by the scheduler management node to determine, based on a resource allocation policy, a target amount of resources to be allocated from the computing memory of the first computing node to the cache system of the first computing node, and obtain a resource allocation result, wherein the resource allocation policy includes at least one of the following: an occupancy parameter of the computing memory, an occupancy parameter of the cache system, and a reserved amount of resources of the computing memory; Receive the resource allocation result sent by the scheduler management node, and allocate the target resource amount in the idle resources of the computing memory to the cache system.

6. The method according to claim 5, characterized in that The determining to expand the capacity of the cache system of the first computing node includes: Determining occupancy parameters of the cache system and release parameters of the cache system, wherein the occupancy parameters of the cache system include: the total amount of resources of the cache system, the amount of occupied resources of the cache system, the amount of idle resources of the cache system, and the average waiting time for resource occupancy of the cache system; and the release parameters of the cache system include: the resource release frequency of the cache system and the resource release result of the cache system; A target resource amount of the cache system to be expanded is determined based on the occupancy parameter of the cache system and the release parameter of the cache system.

7. The method according to claim 5 or 6, characterized in that After allocating the target amount of resources in the idle resources of the computing memory to the cache system, the method further includes: receiving a cache resource release message sent by the scheduler management node, where the cache resource release message is used to instruct: allocating a second resource amount included in the cache system to the computing memory; The second amount of resources included in the cache system is divided into the computing memory.

8. The method according to claim 5 or 6, characterized in that After allocating the target amount of resources in the idle resources of the computing memory to the cache system, the method further includes: Determining, based on an occupancy parameter of the cache system and a release parameter of the cache system, that an amount of occupied resources of the cache system is less than or equal to a preset threshold; The target amount of resources included in the cache system is divided into the computing memory.

9. A cache capacity control device, characterized in that: The cache capacity control device is applied to a scheduler management node included in a cluster, and the cluster also includes multiple computing nodes. The cache capacity control device includes: A first communication module is configured to receive a cache resource request sent by a first computing node; a first processing module, configured to determine, from the computing memory of the first computing node, a target amount of resources to be allocated to the cache system of the first computing node according to the cache resource request and a resource allocation policy, and obtain a resource allocation result, wherein the resource allocation policy includes at least one of the following: an occupancy parameter of the computing memory, an occupancy parameter of the cache system, and a reserved amount of resources of the computing memory; The first communication module is further configured to send the resource allocation result to the first computing node, where the resource allocation result is used to indicate that the target amount of resources in the idle resources of the computing memory is allocated to the cache system.

10. A cache capacity control device, characterized in that: The cache capacity control device is applied to a first computing node among a plurality of computing nodes included in a cluster, wherein the cluster further includes a scheduler management node, and the cache capacity control device includes: A second processing module, configured to determine a capacity of a cache system for expanding the first computing node; a second communication module, configured to send a cache resource request to the scheduler management node, wherein the cache resource request is used by the scheduler management node to determine, based on a resource allocation policy, a target amount of resources to be allocated from the computing memory of the first computing node to the cache system of the first computing node, and obtain a resource allocation result, wherein the resource allocation policy includes at least one of the following: an occupancy parameter of the computing memory, an occupancy parameter of the cache system, and a reserved amount of resources of the computing memory; The second communication module is further configured to receive the resource allocation result sent by the scheduler management node; The second processing module is further configured to allocate the target resource amount in the idle resources of the computing memory to the cache system.

11. An electronic device, characterized in that: The electronic device is a scheduler management node or a first computing node, and the electronic device includes a memory and a processor, and the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-4 or any one of claims 5-8.

12. A cache capacity control system, characterized in that: The cache capacity control system includes multiple computing nodes, including a scheduler management node and a first computing node. The cache capacity control system is used to execute the method according to any one of claims 1 to 4 or any one of claims 5 to 8.

Images