Method, device and storage medium for managing video memory

Abstract

Embodiments of the present invention relate to a method for managing video memory. The method includes: at a board, determining a global video memory region for the board; determining a first video memory region for key-value caching and allocating the first video memory region to a first end of the global video memory region; determining a second video memory region for activation-value caching and allocating the second video memory region to a second end of the global video memory region; and determining unoccupied free video memory regions within the global video memory region based on the first and second video memory regions. By releasing at least a portion of at least one of the first and second video memory regions to the free video memory region, or by converting at least a portion of the free video memory region into at least one of the first and second video memory regions, the current size of at least one of the first and second video memory regions can be dynamically changed, thereby enabling flexible configuration of the first and second video memory regions within the global video memory region.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of computer technology, and more specifically to methods, apparatus and storage media for managing video memory. Background Technology

[0002] Large Language Models (LLMs) typically employ a Transformer architecture and perform inference using methods such as autoregressive inference. Generally, to reduce redundant computations during inference, previous computation results (e.g., keys and values, intermediate activation values, etc.) can be pre-stored in a cache to reduce computational load. However, in existing technologies, the sizes of pre-configured memory regions, such as those for key-value caching and activation value caching, are fixed and cannot be dynamically adjusted, and multiple memory regions are managed separately.

[0003] In summary, the shortcomings of existing solutions for managing video memory are: they cannot coordinate the size of the video memory area used for key-value caching and the size of the video memory area used for activation-value caching, resulting in low video memory utilization. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a method and system for managing video memory, which enables coordinated adjustment of the size of the video memory region used for key-value caching and the size of the video memory region used for activation-value caching, thereby improving video memory utilization and throughput.

[0005] According to a first aspect of the present invention, a method for managing video memory is provided, comprising: at a board, determining a global video memory region relating to the board; determining a first video memory region for key-value caching and allocating the first video memory region to a first end of the global video memory region; determining a second video memory region for activation-value caching and allocating the second video memory region to a second end of the global video memory region; determining an unoccupied empty video memory region in the global video memory region based on the first video memory region and the second video memory region; and changing the current size of at least one of the first and second video memory regions by releasing at least a portion of at least one of the first and second video memory regions to the empty video memory region, or by converting at least a portion of the empty video memory region into at least one of the first and second video memory regions, thereby achieving flexible configuration of the global video memory region.

[0006] In some embodiments, determining the global video memory region for the board includes: determining the size of a weighted video memory region for model weight caching; determining the size of a spare video memory region; and calculating a first predetermined size for the global video memory region based at least on the size of the weighted video memory region and the size of the spare video memory region.

[0007] In some embodiments, the first predetermined size is equal to the total memory size of the board minus the size of the weighted memory region and the size of the spare memory region.

[0008] In some embodiments, determining a first video memory region for key-value caching includes: determining model parameters based on a model used for inference; calculating a video memory block size based on the model parameters; and dividing the first video memory region into blocks based on the video memory block size.

[0009] In some embodiments, determining a second video memory region for caching activation values ​​includes: determining input parameters related to the input, at least based on the input used for inference, the input parameters including batch size and sequence length; determining all activation tensors used for inference according to the input parameters, and determining the video memory size occupied by each activation tensor; and calculating the video memory size occupied by each activation tensor based on a predetermined rule to obtain the size of the second video memory region.

[0010] In some embodiments, the first video memory region has a first video memory type, and the second video memory region has a second video memory type, wherein the first video memory type and the second video memory type are different.

[0011] According to a second aspect of the present invention, a method for accessing video memory managed by the method according to the first aspect of the present invention is provided, comprising: determining whether a currently input parameter is the same as a previously input parameter; in response to the current input parameter being the same as the previously input parameter, determining whether a first video memory region satisfies a threshold condition; and in response to the first video memory region satisfying the threshold condition, accessing the video memory based on the current configuration of a global video memory region.

[0012] In some embodiments, the method of the second aspect of the present invention further includes: in response to the current input parameter being different from the previous input parameter, releasing the current second video memory region and converting the current second video memory region into an empty video memory region to obtain a new empty video memory region; determining the size of the new second video memory region based on the current input parameter; comparing the size of the new second video memory region with the size of the new empty video memory region; and in response to the size of the new second video memory region being less than or equal to the size of the new empty video memory region, allocating the new second video memory region to the second end of the global video memory region.

[0013] In some embodiments, the method of the second aspect of the present invention further includes: determining whether a first memory region includes an unused region in response to the size of the new second memory region being greater than the size of a new empty memory region; releasing unused memory blocks in the unused region in response to the first memory region including an unused region, so as to convert the memory region corresponding to the unused memory blocks into an empty memory region to obtain an expanded new empty memory region; comparing the size of the new second memory region with the size of the expanded new empty memory region; and allocating the new second memory region to the second end of the global memory region in response to the size of the new second memory region being less than or equal to the expanded new empty memory region.

[0014] According to a third aspect of the present invention, an apparatus for managing video memory is provided, comprising: a first video memory manager configured to manage a first video memory region for key-value caching; a second video memory manager configured to manage a second video memory region for activation-value caching; and a global video memory manager configured to: at a board, determine a global video memory region relating to the board; allocate the first video memory region in the global video memory region; and allocate the second video memory region in the global video memory region, wherein the current size of at least one of the first and second video memory regions is changed by releasing at least a portion of at least one of the first and second video memory regions to an unoccupied empty video memory region in the global video memory region, or by converting at least a portion of the empty video memory region into at least one of the first and second video memory regions.

[0015] According to a fourth aspect of the invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the methods of the first and second aspects of the invention.

[0016] According to a fifth aspect of the invention, a computer program product is provided, wherein the computer program product is tangibly stored on a non-transient computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the steps of the methods of the first and second aspects of the invention.

[0017] Compared to existing solutions for managing video memory, the solution provided by this invention can dynamically change the current size of at least one of the first video memory region used for key-value caching and the second video memory region used for activation-value caching, thereby enabling flexible configuration of the first video memory region used for key-value caching and the second video memory region used for activation-value caching in the global video memory region, thereby improving video memory utilization and throughput.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0020] Figure 1 A schematic diagram of a device for managing video memory according to an embodiment of the present invention is shown.

[0021] Figure 2 A flowchart of a method for managing video memory according to an embodiment of the present invention is shown.

[0022] Figure 3 A schematic diagram of the global video memory region according to an embodiment of the present invention is shown.

[0023] Figure 4 A flowchart of a method for accessing video memory according to an embodiment of the present invention is shown. Detailed Implementation

[0024] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] Large Language Models (LLMs) typically employ a Transformer architecture and use inference methods such as autoregressive inference. To reduce redundant computations during inference, key-value caching (KVCache) techniques are often used. This involves pre-storing previous computation results (such as keys and values) in a cache, allowing subsequent inference processes to directly retrieve these results without recalculation, significantly reducing computational load and improving inference efficiency. In this case, a large block of memory, such as GPU memory, needs to be pre-allocated for key-value caching. To better manage the memory used for key-value caching, PageAttention techniques can be further utilized to store contiguous key-value pairs in non-contiguous memory locations, maximizing memory utilization, improving throughput, and avoiding memory waste.

[0027] During the inference process of a large language model, intermediate activation values ​​calculated during the forward pass are stored in GPU memory. Optionally, key-value caching can be used to reduce redundant calculations and improve inference efficiency. Generally, the amount of GPU memory used to store intermediate activation values ​​is related to the size of the model's input data. For example, the longer the sequence length of the model's input, the more GPU memory is required to store intermediate activation values. In other words, when the application scenario of a large language model involves a large amount of long text, a larger storage space needs to be pre-allocated in GPU memory for caching activation values.

[0028] In the above scenario, the large preset storage space for the activation value cache limits the available storage space in video memory for the key-value cache. However, existing video memory management solutions, including the PageAttention technology mentioned above, can only optimize the video memory area used for the key-value cache, but cannot simultaneously optimize the video memory area used for the activation value cache. Therefore, current technology lacks a video memory optimization scheme that can collaboratively adjust the video memory areas used for the key-value cache and the activation value cache.

[0029] Furthermore, in application scenarios involving a mix of long and short texts, especially those with a small amount of long text and a large amount of short text, the size of the storage space preset in the video memory for caching activation values ​​needs to take into account the amount of video memory required for activating long texts. This results in a significant waste of video memory used for caching activation values ​​when performing inference on short texts, leading to low video memory utilization and a decrease in the overall throughput of video memory.

[0030] In summary, the shortcomings of existing solutions for managing video memory are: they cannot coordinate the size of the video memory area used for key-value caching and the size of the video memory area used for activation value caching, and the video memory utilization is low in application scenarios with a mix of long and short texts.

[0031] To at least partially address one or more of the aforementioned problems and other potential issues, an exemplary embodiment of the present invention proposes a scheme for managing video memory. The scheme includes: at a board, determining a global video memory region for the board; determining a first video memory region for key-value caching and allocating the first video memory region to a first end of the global video memory region; determining a second video memory region for activation-value caching and allocating the second video memory region to a second end of the global video memory region; and determining unoccupied empty video memory regions within the global video memory region based on the first and second video memory regions. By releasing at least a portion of at least one of the first and second video memory regions to the empty video memory region, or converting at least a portion of the empty video memory region into at least one of the first and second video memory regions, the current size of at least one of the first and second video memory regions can be dynamically changed, thereby enabling flexible configuration of the first video memory region for key-value caching and the second video memory region for activation-value caching within the global video memory region.

[0032] The following will combine Figures 1 to 3 A scheme for managing video memory according to an embodiment of the present invention is described in detail.

[0033] Figure 1 A schematic diagram of a device 100 for managing video memory according to an embodiment of the present invention is shown. It should be understood that the device 100 may also include additional devices not shown and / or one or more of the shown devices may be integrated onto one or more devices, and the scope of the invention is not limited in this respect.

[0034] like Figure 1 As shown, device 100 may include: a first video memory manager 101, a second video memory manager 102, and a global video memory manager 103.

[0035] Regarding the first video memory manager 101, it can be configured to manage a first video memory region used for key-value cache (KVCache). According to embodiments of the invention, the first video memory manager 101 can be configured, for example, to request additional storage space from the global video memory manager 103 for use in the key-value cache. In yet other embodiments, the first video memory manager 101 can also be configured to release unused storage space from the global video memory manager 103.

[0036] Regarding the second memory manager 102, it can be configured to manage a second memory region used for the activation value cache. According to an embodiment of the present invention, the second memory manager 102 can be configured, for example, to calculate the size of the second memory region used for the activation value cache.

[0037] Regarding the global video memory manager 103, it can be configured to: determine the global video memory region for the board at the board location; allocate a first video memory region within the global video memory region; and allocate a second video memory region within the global video memory region. The "board" here can include, but is not limited to: a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processor (TPU), a neural network processor (NPU), etc.

[0038] According to an embodiment of the present invention, the device 100 may be further configured to: release at least a portion of at least one of the first and second video memory regions to an empty video memory region via a first video memory manager 101, a second video memory manager 102, and a global video memory manager 103, or convert at least a portion of the empty video memory region into at least one of the first and second video memory regions, thereby changing the current size of at least one of the first and second video memory regions, enabling dynamic management of the first and second video memory regions, and achieving flexible configuration of the global video memory region. The following will... Figure 2 Detailed description Figure 1 How can device 100 achieve flexible configuration of the global video memory area?

[0039] Figure 2 A flowchart of a method 200 for managing video memory according to an embodiment of the present invention is shown. It should be understood that method 200 may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the invention is not limited in this respect.

[0040] In step 201, at the board, the global video memory manager 103 determines the global video memory area for the board.

[0041] Regarding the global video memory region, according to embodiments of the present invention, it may refer to a pre-allocated block of storage space within the video memory of the board for subsequent storage of key values ​​and activation values. Specifically, the global video memory region may include: a first video memory region for key value caching, a second video memory region for activation value caching, and an empty video memory region, wherein the empty video memory region may be configured to provide additional storage space to the first video memory region and / or the second video memory region.

[0042] Typically, during the inference process of a large language model, in addition to key values ​​and activation values, data such as model weights are also required to be stored in the graphics memory of the board. Therefore, in some embodiments of the present invention, the storage space in the graphics memory of the board that can be used as a global graphics memory area and its size can be determined at least based on the storage space used to store model weights.

[0043] Specifically, according to an embodiment of the present invention, determining the global video memory region of the board may further include: determining the size of the weight video memory region for model weight caching; determining the size of the spare video memory region; and calculating a first predetermined size for the global video memory region based at least on the size of the weight video memory region and the size of the spare video memory region.

[0044] Regarding the weight memory area used for model weight caching, as mentioned above, it can refer to the storage space used to store model weights.

[0045] Regarding the spare video memory area, it can refer to the size of the storage space in video memory that serves as a buffer. It should be understood that in practice, it is usually necessary to reserve a small amount of spare storage space in the video memory of the board as a buffer to buffer data such as input and / or output.

[0046] According to embodiments of the present invention, other memory regions in the graphics card's memory besides the weighted memory region and the spare memory region can be allocated as global memory regions for storing key values ​​and activation values. Therefore, in embodiments of the present invention, the size of the memory available for use as global memory regions in the graphics card's memory can be calculated by subtracting the size of the weighted memory region and the size of the spare memory region from the total memory size of the graphics card. That is, the first predetermined size of the global memory region can be equal to the total memory size of the graphics card minus the size of the weighted memory region and the size of the spare memory region.

[0047] As can be seen from the above, in the solution for managing video memory provided by the present invention, the global video memory manager 103 can determine a whole block of storage space with a first predetermined size in the video memory of the board, such as according to a first predetermined size for the global video memory region calculated by the global video memory manager 103, as the global video memory region of the board.

[0048] According to the inventive concept of the present invention, the global memory manager 103 can also be initialized based on a determined global memory region having a first predetermined size. Specifically, the global memory manager 103 can record the initial address and size of the global memory region.

[0049] In step 202, the global video memory manager 103 determines the first video memory region for key-value caching and allocates the first video memory region to the first end of the global video memory region.

[0050] According to the inventive concept of the present invention, the size and location of the first video memory region can be determined by the global video memory manager 103, wherein the first video memory region is configured to store key values.

[0051] by Figure 3 Taking the global memory region 300 shown in the figure as an example, Figure 3 As shown, the global memory manager 103 can determine the first memory region 301 for key-value caching and allocate the first memory region 301 to one end of the global memory region 300, such as the initial position of the global memory region 300.

[0052] The size of the first video memory region 301 used for key-value caching can be determined based on empirical values. In some embodiments, the size of the first video memory region 301 used for key-value caching can also be determined by pre-inputting a specified batch size of data into the model and then performing calculations.

[0053] According to the inventive concept of the present invention, the first video memory region 301 can then be divided into multiple video memory blocks by the first video memory manager 101, so that the first video memory region 301 can be managed based on the video memory blocks (e.g., requesting additional storage space, releasing excess storage space, etc.).

[0054] Regarding video memory blocks, they can refer to video memory sub-regions with the same storage space. According to an embodiment of the present invention, the first video memory manager 101 can first determine the size of the video memory block, and then divide the first video memory region 301 into blocks based on the determined size of the video memory block. Specifically, in an embodiment of the present invention, the first video memory manager 101 can further determine model parameters based on the model used for inference; calculate the video memory block size based on the model parameters; and divide the first video memory region 301 into blocks based on the video memory block size.

[0055] Regarding model parameters, according to embodiments of the present invention, they may include: the number of heads (num_heads) and the size of each head (head_size) of the model's attention mechanism, the number of layers (num_layers) of the model, and the size corresponding to the model's accuracy (datatype_size).

[0056] Assuming that each memory block can store n (n is an integer greater than 0) key-value caches of tokens, and denoted as "slot_size", according to some embodiments of the present invention, combined with the multiple model parameters listed above, the size of each memory block (block_size) can be calculated according to the following formula (1).

[0057] (1)

[0058] Then, based on the calculated size (block_size) of each memory block, the first memory manager 101 can divide the first memory region 301 into blocks to obtain N memory blocks of the same size, as shown in the following example. Figure 3 As shown, N is equal to the size of the first video memory region 301 divided by the size of each video memory block (block_size).

[0059] In step 203, the global memory manager 103 determines a second memory region for activating the value cache and allocates the second memory region to the second end of the global memory region.

[0060] According to the inventive concept of the present invention, the size and location of the second video memory region can be determined by the global video memory manager 103, wherein the second video memory region is configured to store activation values.

[0061] Still with Figure 3 Taking the global memory region 300 shown as an example, the global memory manager 103 can determine the second memory region 302 used for the activation value cache and allocate the second memory region 302 to the other end of the global memory region 300, such as... Figure 3 At the end of the global video memory region 300.

[0062] The size of the second memory region 302 used for activating value caching can be related to the total number of terms involved in model inference, that is, to the batch size and sequence length of the data to be input into the model. Generally, the larger the batch size, the larger the size of the second memory region 302 used for activating value caching; the larger the sequence length, the larger the size of the second memory region 302 used for activating value caching.

[0063] According to the inventive concept of the present invention, the size of the determined second video memory region 302 for activation value caching can vary with the inputs fed into the model for inference. For example, in an embodiment of the present invention, determining the second video memory region 302 for activation value caching may include: determining input parameters related to the input by the second video memory manager 102, at least based on the inputs used for inference; determining all activation tensors used for inference according to the input parameters, and determining the video memory size occupied by each activation tensor; and calculating the video memory size occupied by each activation tensor based on a predetermined rule to obtain the size of the second video memory region 302.

[0064] Regarding input parameters, these can include things like batch size and sequence length.

[0065] Regarding the predetermined rules, these can refer to rules such as summation rules or reuse rules. According to some embodiments of the present invention, the memory size occupied by each active tensor can be summed to obtain the size of the second memory region 302. According to further embodiments of the present invention, the memory size occupied by each active tensor can be calculated based on reuse rules, that is, based on the lifetime of each active tensor, the memory size occupied by each active tensor is calculated according to the memory reuse rules to obtain the size of the second memory region 302.

[0066] Regarding the reuse rules, according to embodiments of the present invention, they may be, for example, memory reuse rules based on an implicit free list.

[0067] Taking the computation of graphs in large language model reasoning as an example, the output data and temporary data of each node in the graph to be computed can be regarded as activation values. As the input parameters change, that is, as the batch size and sequence length corresponding to the current input change, the size of the output data and temporary data of each node also changes accordingly. To address this characteristic, the inventive concept of this invention further includes: whenever the input changes, that is, the batch size or sequence length of the input changes, recalculating the size of the activation values ​​corresponding to each node, and based on this, redetermining the size of the second video memory region used for caching activation values ​​according to reuse rules.

[0068] According to an embodiment of the present invention, the second memory manager 102 can determine all active tensors used for inference and the memory size occupied by each active tensor, based at least on the batch size and sequence length of the input used for inference, wherein the memory size occupied by each active tensor is not necessarily the same. Then, the second memory manager 102 can perform a calculation based on predetermined rules (e.g., summation or calculation based on the reuse rules of an implicit free list) on the memory size occupied by each active tensor to calculate the total memory size occupied by all active tensors, thereby obtaining the size of the second memory region 302. Based on this, the global memory manager 103 can allocate the second memory region 302 to the end of the global memory region 300 as described above, based on the obtained size of the second memory region 302.

[0069] Furthermore, in some embodiments, by using reuse rules to determine the size of the second video memory region used for activation value caching, the video memory occupied by the activation value of a node in the graph of the large language model inference can be released and used by the activation values ​​of other nodes when the computation of that node is completed and the data computed by that node will not be used by other nodes in the graph. Thus, the size of the video memory occupied by the second video memory region used for activation value caching can be further reduced, allowing more video memory space to be used for key-value caching.

[0070] Furthermore, according to some embodiments of the present invention, the determination of the size of the second video memory region used for the activation value cache and the allocation of the second video memory region in the global video memory region only occur when the input parameters change, that is, when the pre-filling and decoding stages of inference are switched. Therefore, the time for calculating the size of the second video memory region used for the activation value cache and allocating the second video memory region can be significantly reduced.

[0071] In step 204, the global video memory manager 103 determines the unoccupied empty video memory area in the global video memory area based on the first video memory area and the second video memory area.

[0072] According to the inventive concept of the present invention, an empty video memory region can be configured in the global video memory region to enable flexible configuration of the first and second video memory regions in the global video memory region. An empty video memory region can refer to storage space in the global video memory region that is neither allocated for storing key values ​​nor for storing activation values.

[0073] According to an embodiment of the present invention, the global video memory manager 103 can determine the area outside the first video memory area and the second video memory area, that is, the part of the unoccupied global video memory area, as an empty video memory area.

[0074] by Figure 3 The global memory region 300 shown in the figure is as follows: Figure 3 As shown, the area in the global video memory region 300 other than the first video memory region 301 and the second video memory region 302, that is, the video memory region between the first video memory region 301 and the second video memory region 302, can be regarded as the empty video memory region 303.

[0075] In order to balance the size of the memory region configured in the global memory region for key value caching and the size of the memory region configured for activation value caching, and to achieve dynamic adjustment of both, according to the inventive concept of the present invention, while keeping the size of the global memory region 300 unchanged, the relative sizes of the first memory region 301, the second memory region 302 and the empty memory region 303 can be changed to achieve flexible configuration of the global memory region 300.

[0076] Specifically, in step 205, at least a portion of at least one of the first video memory region 301 and the second video memory region 302 is released to the empty video memory region 303, or at least a portion of the empty video memory region 303 is converted into at least one of the first video memory region 301 and the second video memory region 302, thereby changing the current size of at least one of the first video memory region 301 and the second video memory region 302, so as to achieve flexible configuration of the global video memory region 300.

[0077] Regarding releasing at least a portion of at least one of the first video memory region 301 and the second video memory region 302 to the empty video memory region 303, it can mean: releasing at least a portion of the first video memory region 301 to the empty video memory region 303 to reduce the size of the first video memory region 301 and increase the size of the empty video memory region 303, while keeping the size of the second video memory region 302 unchanged; or, releasing all of the second video memory region 302 to the empty video memory region 303 to reduce the size of the second video memory region 302 and increase the size of the empty video memory region 303, while keeping the size of the first video memory region 301 unchanged.

[0078] According to some embodiments of the present invention, at least a portion of the first video memory region 301 adjacent to the empty video memory region 303 can be released to the empty video memory region 303 by the first video memory manager 101. According to still some embodiments of the present invention, the entire second video memory region 302 can be released to the empty video memory region 303 by the second video memory manager 102.

[0079] Regarding converting at least a portion of the empty video memory region 303 into at least one of the first video memory region 301 and the second video memory region 302, it may mean: converting at least a portion of the empty video memory region 303 into the first video memory region 301 to increase the size of the first video memory region 301 and decrease the size of the empty video memory region 303, while keeping the second video memory region 302 unchanged; or, converting at least a portion of the empty video memory region 303 into the second video memory region 302, while keeping the size of the first video memory region 301 unchanged.

[0080] According to some embodiments of the present invention, the first video memory manager 101 may send a request for additional storage space to the global video memory manager 103, so that the global video memory manager 103 may convert at least a portion of the empty video memory region 303 adjacent to the first video memory region 301 into the first video memory region 301. According to further embodiments of the present invention, the second video memory manager 102 may determine the size of the empty video memory region 303 to be converted into the second video memory region 302, and the global video memory manager 103 may convert the region at the end of the empty video memory region 303 into the second video memory region 302 based on the determined size of the empty video memory region 303 to be converted into the second video memory region 302.

[0081] As can be seen from the above, the method for managing video memory provided by the embodiments of the present invention can take into account both the video memory area used for key-value caching and the video memory area used for activation value caching, and dynamically adjust the relative size of the video memory area used for key-value caching and the video memory area used for activation value caching by means of empty video memory area, so as to improve video memory utilization and throughput.

[0082] Furthermore, as can be seen from the above, the device for managing video memory provided by the present invention can realize two-level management of video memory at the board, that is, by the device's first video memory manager (such as...) Figure 1 First memory manager 101) and second memory manager (such as Figure 1 The second video memory manager 102) respectively implements the management of the first video memory region (such as... Figure 3 First memory region 301) and second memory region (such as Figure 3 The second video memory region 302 is managed by the global video memory manager (such as...). Figure 1 The global video memory manager 103 enables the management of global video memory regions (such as...) Figure 3 The global video memory area (300) allows for more flexible management of the first and second video memory areas, improving video memory utilization. At the same time, it reduces the number of times memory space is requested and / or released when adjusting the size of the first and second video memory areas, thus improving efficiency.

[0083] Furthermore, in the technical solutions provided in the embodiments of the present invention, the first video memory region and the second video memory region are managed by different video memory managers. That is, the video memory region used for key-value caching and the video memory region used for activation-value caching are managed separately, allowing different management methods to be used for the two regions depending on the actual usage scenario. For example, in some embodiments of the present invention, a first management method can be used to manage the first video memory region, and a second management method can be used to manage the second video memory region. In one example, a paging attention mechanism can be used to manage the first video memory region, and a dedicated method different from the paging attention mechanism can be used to manage the second video memory region. Therefore, the proposed solution for managing video memory can also achieve the management of video memory regions with different video memory types, namely, those used for key-value caching and those used for activation-value caching.

[0084] According to an embodiment of the present invention, a first video memory region (such as...) is used for key-value caching. Figure 3 The first video memory region 301) has a first video memory type, and the second video memory region (such as the activation value cache) is used for the second video memory region. Figure 3 The second video memory region 302) has a second video memory type that is different from the first video memory type. For example, the first video memory type of the first video memory region used for key-value caching can be NUMA (Non-uniform memory access) video memory type, and the second video memory type of the second video memory region used for activation value caching can be UMA (Uniform memory access) video memory type.

[0085] The following will combine Figure 4This section describes in detail how to access the video memory managed by the method for managing video memory provided according to an embodiment of the present invention.

[0086] Figure 4 A flowchart illustrating a method 400 for accessing video memory according to an embodiment of the present invention is shown. According to an embodiment of the present invention, method 400 may be provided by, for example... Figure 1 The device 100 performs the procedure. It should be understood that method 400 may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the invention is not limited in this respect.

[0087] In step 401, at the board, the global video memory region for the board is determined.

[0088] According to an embodiment of the present invention, the global memory manager 103 can determine the global memory region of the board at the board location. For example, the global memory manager 103 can determine all memory regions at the board location other than those used for storing weights and those used as reserves as the global memory region of the board.

[0089] For details regarding step 401, please refer to the description of step 201 above; it will not be repeated here.

[0090] In step 402, a first video memory region for key-value caching is determined and allocated to the first end of the global video memory region.

[0091] According to an embodiment of the present invention, the global memory manager 103 may determine a first memory region for key-value caching and allocate the first memory region to the first end of the global memory region. For example, the global memory manager 103 may determine a first memory region for key-value caching and allocate the first memory region to the initial position of the global memory region.

[0092] For details regarding step 402, please refer to the description of step 202 above; it will not be repeated here.

[0093] In step 403, a second video memory region for activating the value cache is determined, and the second video memory region is allocated to the second end of the global video memory region.

[0094] According to an embodiment of the present invention, the global memory manager 103 may determine a second memory region for activating the value cache and allocate the second memory region to the second end of the global memory region. For example, the global memory manager 103 may determine a second memory region for activating the value cache and allocate the second memory region to the end of the global memory region.

[0095] For details regarding step 403, please refer to the description of step 203 above; it will not be repeated here.

[0096] In step 404, based on the first video memory region and the second video memory region, the unoccupied empty video memory region in the global video memory region is determined.

[0097] According to an embodiment of the present invention, the global memory manager 103 can determine the unoccupied empty memory area in the global memory region based on the first memory region and the second memory region. For example, the global memory manager 103 can determine the unoccupied global memory area between the first memory region and the second memory region as an empty memory area.

[0098] The description of step 404 is the same as that of step 204 above, and will not be repeated here.

[0099] Based on steps 401 to 404 above, the initial configuration of the global video memory region can be completed at the board. It should be understood that, according to the inventive concept of the present invention, the initial configuration can be dynamically changed during the subsequent model inference process.

[0100] In step 405, it is determined whether the current input parameter is the same as the previous input parameter.

[0101] According to the inventive concept of the present invention, since the size of the second memory region required for reasoning in a large language model is related to the input for reasoning, the size of the required second memory region can be determined to have changed during each reasoning operation by judging whether the input used for reasoning has changed. For example, the shape of the current input can be determined to have changed compared to the previous input by judging whether the shape of the input used for reasoning has changed. In some embodiments of the present invention, the shape of the input used for reasoning can be determined to have changed by comparing the input parameters of the current input with the input parameters of the previous input.

[0102] If the current input parameter is the same as the previous input parameter, it indicates that the shape of the input used for inference has not changed. Therefore, the size of the second memory region corresponding to the current inference is the same as the size of the second memory region corresponding to the previous inference. In other words, the current inference can be performed based on the second memory region determined in the previous inference. In this case, it can be further determined whether the size of the first memory region meets the requirements for performing the current inference, to determine whether inference can also be performed based on the current first memory region. Therefore, according to an embodiment of the present invention, in response to determining at step 405 that the current input parameter is the same as the previous input parameter, the process proceeds to step 406 to determine whether the first memory region meets the threshold condition.

[0103] Regarding the threshold condition, it can refer to a condition that indicates that the first video memory region has sufficient storage space. According to embodiments of the present invention, the threshold condition may include, for example, that the currently available storage space in the first video memory region is sufficient to store the key value of the current inference.

[0104] According to an embodiment of the present invention, the first video memory manager 101 can determine whether the first video memory region meets the threshold condition. For example, the first video memory manager 101 can determine whether the currently available storage space of the first video memory region is sufficient to store the key value of the current inference.

[0105] If the first video memory region meets the threshold condition, meaning that the currently available storage space in the first video memory region is sufficient to store the key value for the current inference, it indicates that the current inference can be completed based on the first and second video memory regions configured in the current global video memory region. Therefore, in response to determining in step 406 that the first video memory region meets the threshold condition, the process proceeds to step 407, where video memory is accessed based on the current configuration of the global video memory region.

[0106] If the first video memory region does not meet the threshold condition, meaning the currently available storage space of the first video memory region is insufficient to store the key value for the current inference, then it is necessary to convert at least a portion of the empty video memory region into the first video memory region. By expanding the available storage space of the first video memory region, the current inference can be completed based on the expanded available storage space of the first video memory region. Therefore, according to an embodiment of the present invention, in response to determining at step 406 that the first video memory region does not meet the threshold condition, the process proceeds to step 408, where at least a portion of the empty video memory region is converted into the first video memory region.

[0107] According to embodiments of the present invention, the global memory manager 103 can convert at least a portion of the empty memory region into a first memory region. For example, in some embodiments, the first memory manager 101 issues a request for additional storage space to the global memory manager 103 at the granularity of memory block size, wherein the request for additional storage space may include information indicating the size of the additional storage space to be requested (e.g., information related to the number of memory blocks, etc.). In response to receiving this request, the global memory manager 103 converts at least a portion (or even all of the empty memory region) of the empty memory region adjacent to the first memory region into the first memory region, thereby obtaining an expanded first memory region. Thus, the available storage space of the expanded first memory region includes at least a portion of the first memory region converted from the empty memory region, so that the available storage space of the expanded first memory region can meet the needs of completing the current inference. Based on this, the process can then proceed to step 407, where memory is accessed based on the current configuration of the global memory region, wherein the current configuration of the global memory region includes the expanded first memory region.

[0108] In some embodiments, if the available storage space of the expanded first memory region obtained by converting all empty video memory regions into the first memory region is still insufficient to meet the requirements for completing the current inference, given that the current global video memory region is completely filled by the first and second video memory regions, a swap operation can be performed, for example, to swap the key values ​​of lower priority statements stored outside the current board to other boards (such as the central processing unit (CPU)) so that some storage space in the current board's global video memory region can be freed up for storing the key values ​​of higher priority statements.

[0109] Returning to step 405, if the current input parameters differ from the previous input parameters, it indicates a change in the shape of the input used for inference. According to the inventive concept of the present invention, it is necessary to redetermine the size of the second video memory region used for activating the value cache. Therefore, according to an embodiment of the present invention, in response to determining at step 405 that the current input parameters differ from the previous input parameters, proceed to step 409, release the current second video memory region, determine a new second video memory region, and allocate the new second video memory region to the second end of the global video memory region.

[0110] According to an embodiment of the present invention, the global memory manager 103 can release the current second memory region, determine a new second memory region, and allocate the new second memory region to the second end of the global memory region. For example, the global memory manager 103 can release the current second memory region to convert the entire current second memory region into an empty memory region; based on the currently input parameters, the global memory manager 103 redetermines the second memory region (which can be denoted as the "new second memory region") for the activation value cache, and allocates the new second memory region to the second end (i.e., the end position) of the global memory region, that is, the global memory manager 103 allocates the new second memory region to the empty memory region and away from the first memory region. For the specific steps of determining the new second memory region for the activation value cache, please refer to the section on... Figure 2 The description of step 203 will not be repeated here.

[0111] However, in some embodiments, before allocating the new second video memory region to the second end of the global video memory region, in order to ensure that there is sufficient storage space in the global video memory region, the video memory size required by the new second video memory region can be compared with the available storage space in the global video memory region excluding the already configured first video memory region. Therefore, for these embodiments, step 409 above may further include: releasing the current second video memory region to convert the current second video memory region into an empty video memory region to obtain a new empty video memory region; determining the size of the new second video memory region based on the currently input parameters; comparing the size of the new second video memory region with the size of the new empty video memory region; and allocating the new second video memory region to the second end of the global video memory region in response to the new second video memory region being less than or equal to the size of the new empty video memory region.

[0112] For example, the global memory manager 103 releases the current second memory region to convert it into an empty memory region, thus obtaining a new empty memory region. At this time, the global memory region only includes the first memory region and the new empty memory region. Based on this, the second memory manager 102 can redetermine the size of the new second memory region based on the input parameters of the current input (including the batch size and sequence length of the current input), for example, by using the input parameters of the current input to redetermine all activation tensors used for inference and the memory size occupied by each activation tensor. Then, the global memory manager 103 can compare the size of the new second memory region with the size of the new empty memory region to determine whether the size of the new second memory region is less than or equal to the size of the new empty memory region.

[0113] In some embodiments, if the size of the new second video memory region is less than or equal to the size of the new empty video memory region, it means that there is enough storage space in the current global video memory region to configure the new second video memory region. The global video memory manager 103 can allocate the new second video memory region to the second end of the global video memory region to realize the reconfiguration of the second video memory region of the global video memory region.

[0114] In some other embodiments, if the size of the new second video memory region is larger than the size of the new empty video memory region, it indicates that there is insufficient storage space in the current global video memory region to configure the new second video memory region. In this case, unused storage space in the first video memory region can be released to ensure that there is sufficient storage space in the global video memory region to configure the new second video memory region. Specifically, for these embodiments, it is further determined whether the first video memory region includes unused regions; in response to the first video memory region including unused regions, unused video memory blocks in the unused regions are released to convert the video memory regions corresponding to the unused video memory blocks into empty video memory regions to obtain expanded new empty video memory regions; the size of the new second video memory region is compared with the size of the expanded new empty video memory region; and in response to the size of the new second video memory region being less than or equal to the expanded new empty video memory region, the new second video memory region is allocated to the second end of the global video memory region.

[0115] For example, the first video memory manager 101 can determine whether the first video memory region includes an unused region; in response to the first video memory region including an unused region, the global video memory manager 103 releases the unused video memory blocks in the unused region of the first video memory region to convert the video memory region corresponding to the unused video memory blocks in the first video memory region into an empty video memory region, and obtains an expanded new empty video memory region; the global video memory manager 103 compares the size of the new second video memory region with the size of the expanded new empty video memory region; and in response to the size of the new second video memory region being less than or equal to the expanded new empty video memory region, the global video memory manager 103 allocates the new second video memory region to the second end of the global video memory region.

[0116] Regarding determining whether the first video memory region includes unused areas, it can be determined by the first video memory manager 101 whether the end of the first video memory region (i.e., the position adjacent to the empty video memory region) includes unused video memory blocks.

[0117] Regarding the release of unused video memory blocks in unused areas, it can be done by the first video memory manager 101 releasing some or all of the unused video memory blocks in the unused areas.

[0118] However, in some embodiments, if the first video memory region does not include unused areas, or the size of the new second video memory region is larger than the expanded new empty video memory region, that is, the global video memory region is currently insufficient to accommodate the new second video memory region, according to embodiments of the present invention, the size of the second video memory region corresponding to the current inference can be reduced by means such as reducing the input size used for the current inference (such as the number of scheduled statements), or the current statement can be delayed in scheduling.

[0119] As can be seen from the above, the method for accessing video memory provided by the present invention can flexibly configure the global video memory region at the board based on the input used for inference, enabling coordinated adjustment of the size of the first video memory region used for key-value caching and the second video memory region used for activation value caching within the global video memory region, thereby improving video memory utilization and overall throughput. In particular, when the input used for model inference includes both long and short texts, the solution provided by the present invention can significantly improve video memory utilization and overall throughput.

[0120] Combination Figure 1 The device 100 is described below as an example of a scheme for managing and accessing video memory according to the present invention.

[0121] Step 1: During the inference initialization phase of the large language model, device 100 determines all video memory areas on the board, excluding those used for storing weights and those reserved, as the global video memory area for that board, and initializes the global video memory manager 103 with the determined global video memory area. According to an embodiment of the present invention, the global video memory manager 103 can be configured to manage unoccupied storage space within the global video memory area. For example, the global video memory manager 103 can be configured to record the initial address of the global video memory area, the offset of the unoccupied storage space within the global video memory area, and its size. At this time, the offset of the unoccupied storage space within the global video memory area is 0, and the size of the unoccupied storage space within the global video memory area is equal to the size of the determined entire global video memory area.

[0122] Step Two: The global memory manager 103 determines a first memory region for key-value caching within the global memory region determined in Step One, and the first memory manager 101 divides the first memory region into multiple memory blocks of equal size. At this time, the offset of the unoccupied storage space in the global memory region is equal to the size of the first memory region, and the size of the unoccupied storage space in the global memory region is equal to the size of the entire global memory region minus the size of the first memory region.

[0123] Step 3: The second video memory manager 102, using a memory reuse method, traverses the computation graph once to calculate the relative offset and size of the storage space occupied by each active tensor in the computation graph, and calculates the size of the second video memory region used for caching active values ​​based on the storage space occupied by each active tensor. The global video memory manager 103 can then allocate the second video memory region to the end position of the global video memory region determined in step one. At this time, the offset value of the unoccupied storage space in the global video memory region is equal to the size of the first video memory region, and the size of the unoccupied storage space in the global video memory region is equal to the size of the entire global video memory region minus the size of the first video memory region, and then minus the size of the second video memory region.

[0124] Step 4: For each inference step, check if the shape of the input used for that inference (i.e., the input parameters) has changed. If the shape of the input has changed, proceed to Step 5; otherwise, proceed to Step 11.

[0125] Step 5: In response to a change in the shape of the input for the current inference, the global memory manager 103 releases the original second memory region, converting the second memory region corresponding to the previous inference into an empty memory region. Based on this, the relative offset positions and sizes of the storage spaces occupied by all activation tensors in the computation graph are recalculated using the shape of the input for the current inference, and the size of the new second memory region used for caching activation values ​​is calculated.

[0126] Step Six: The Global Memory Manager 103 determines whether the size of the unoccupied storage space in the global memory region is greater than or equal to the size of the new second memory region calculated in Step Five. If the size of the unoccupied storage space in the global memory region is greater than or equal to the size of the new second memory region, proceed to Step Ten; if the size of the unoccupied storage space in the global memory region is less than the size of the new second memory region, proceed to Step Seven.

[0127] Step 7: If the global memory manager 103 determines that the size of the unused storage space in the global memory region is smaller than the size of the new second memory region, the first memory manager 101 releases the unused memory block at the end of the first memory region into the global memory manager 103, that is, converts the unused memory block at the end of the first memory region into an empty memory region to meet the memory size requirement of the new second memory region.

[0128] Step 8: After the first video memory manager 101 releases the unused video memory blocks at the end of the first video memory region, it determines whether the unoccupied storage space in the global video memory region meets the requirements for configuring a new second video memory region. If the unoccupied storage space in the global video memory region can meet the requirements for configuring a new second video memory region, proceed to step 10; if the unoccupied storage space in the global video memory region cannot meet the requirements for configuring a new second video memory region, proceed to step 9.

[0129] Step Nine: After the first memory manager 101 releases the unused memory block at the end of the first memory region, the unoccupied storage space in the global memory region is still insufficient to meet the requirements for configuring a new second memory region. Therefore, the number of statements scheduled for this inference operation can be reduced to ensure sufficient memory availability. In some cases, if even reducing the number to only one statement for this inference operation is insufficient to configure a new second memory region, then scheduling that statement can be postponed.

[0130] Step 10: The global memory manager 103 allocates a new second memory region at the end of the global memory region, which is managed by the second memory manager 102. The physical address of the activated tensor is equal to the initial address of the global memory region recorded in the global memory manager, plus the offset of the unoccupied storage space in the global memory region, plus the size of the unoccupied storage space in the global memory region, and plus the relative offset of the activated tensor in the second memory manager 102.

[0131] Step 11: The first video memory manager 101 determines whether there is enough unused storage space in the first video memory region. If there is enough unused storage space in the first video memory region, proceed to step 12; if there is not enough unused storage space in the first video memory region, proceed to step 15.

[0132] Step 12: The first video memory manager 101 requests additional storage space from the global video memory manager 103 at the size of video memory blocks to expand the unused storage space in the first video memory region to meet usage requirements. Correspondingly, the unoccupied storage space in the global video memory region managed by the global video memory manager 103 will decrease.

[0133] Step Thirteen: The first memory manager 101 determines whether the additional storage space it has allocated is sufficient to meet the usage requirements. If the additional storage space allocated by the first memory manager 101 is sufficient to meet the usage requirements, proceed to Step Fifteen; if the additional storage space allocated by the first memory manager 101 is insufficient to meet the usage requirements, proceed to Step Fourteen.

[0134] Step Fourteen: When the first video memory manager 101 requests all the unused storage space in the global video memory region but still cannot meet the usage demand, since the storage space of the entire global video memory region is full and there is no extra storage space available, in this case, a swap operation can be performed to swap the first video memory region occupied by lower priority statements to a location such as the CPU, so as to free up some storage space in the first video memory region for higher priority statements to use.

[0135] Step 15: When the current inference is complete, check if a new inference request has been received. If a new inference request has been received, return to Step 4; if no new inference request has been received, proceed to Step 16.

[0136] Step Sixteen: Check if a termination condition has been encountered to end the reasoning. If the reasoning ends, proceed to Step Seventeen; if the reasoning continues, return to Step Four.

[0137] Step 16: Deinitialize the first video memory manager 101, the second video memory manager 102, and the global video memory manager 103, and release the global video memory area on the board.

[0138] This invention can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0139] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0140] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0143] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0144] The above are merely optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for managing video memory, characterized in that, include: At the board location, determine the global video memory area for the board; A first video memory region for key-value caching is determined, and the first video memory region is allocated to the first end of the global video memory region; A second video memory region for activating the value cache is determined, and the second video memory region is allocated to the second end of the global video memory region; Based on the first video memory region and the second video memory region, determine the unoccupied empty video memory region in the global video memory region; as well as By releasing at least a portion of at least one of the first and second video memory regions into the empty video memory region, or by converting at least a portion of the empty video memory region into at least one of the first and second video memory regions, the current size of at least one of the first and second video memory regions is changed, thereby enabling flexible configuration of the global video memory region. The second video memory region used for activating value caching includes: Based at least on the input used for reasoning, determine input parameters related to the input, including batch size and sequence length; Based on the input parameters, determine all activation tensors used for inference, and determine the memory size occupied by each activation tensor; and The size of the second memory region is obtained by calculating the memory size occupied by each activated tensor based on a predetermined rule. The method further includes: in response to the fact that the current input parameter is different from the previous input parameter, releasing the current second video memory region and converting the current second video memory region into the empty video memory region to obtain a new empty video memory region.

2. The method according to claim 1, characterized in that, Determining the global video memory area for the aforementioned board includes: Determine the size of the weight memory region used for model weight caching; Determine the size of the spare video memory area; and A first predetermined size for the global memory region is calculated based at least on the size of the weighted memory region and the size of the spare memory region.

3. The method according to claim 2, characterized in that, The first predetermined size is equal to the total video memory size of the board minus the size of the weighted video memory area and the size of the spare video memory area.

4. The method according to claim 1, characterized in that, The first video memory region to be used for key-value caching includes: Based on the model used for reasoning, determine the model parameters; Based on the model parameters, the memory block size is calculated; and Based on the size of the video memory block, the first video memory region is divided into blocks.

5. The method according to claim 1, characterized in that, The first video memory region has a first video memory type, and the second video memory region has a second video memory type, wherein the first video memory type and the second video memory type are different.

6. A method for accessing video memory managed by the method according to any one of claims 1 to 5, characterized in that, include: Determine whether the current input parameter is the same as the previous input parameter; In response to the fact that the current input parameter is the same as the previous input parameter, it is determined whether the first video memory region meets the threshold condition; In response to the first video memory region meeting the threshold condition, the video memory is accessed based on the current configuration of the global video memory region.

7. The method according to claim 6, characterized in that, Also includes: Based on the input parameters of the current input, determine the size of the new second video memory region; Compare the size of the new second video memory region with the size of the new empty video memory region; as well as In response to the fact that the size of the new second video memory region is less than or equal to the size of the new empty video memory region, the new second video memory region is allocated to the second end of the global video memory region.

8. The method according to claim 7, characterized in that, Also includes: In response to the fact that the size of the new second video memory region is greater than the size of the new empty video memory region, it is determined whether the first video memory region includes unused areas; In response to the first video memory region including an unused region, unused video memory blocks in the unused region are released to convert the video memory region corresponding to the unused video memory blocks into the empty video memory region, thereby obtaining an expanded new empty video memory region; Compare the size of the new second video memory region with the size of the expanded new empty video memory region; as well as In response to the fact that the size of the new second video memory region is less than or equal to the expanded new empty video memory region, the new second video memory region is allocated to the second end of the global video memory region.

9. A device for managing video memory, characterized in that, include: The first video memory manager is configured to manage the first video memory region used for key-value caching; The second video memory manager is configured to manage a second video memory region used for activating value caching. as well as The global video memory manager is configured to be used for: At the board location, determine the global video memory area for the board; Allocate the first video memory region within the global video memory region; and Allocate the second video memory region within the global video memory region. Specifically, by releasing at least a portion of at least one of the first video memory region and the second video memory region to an unoccupied empty video memory region in the global video memory region, or by converting at least a portion of the empty video memory region into at least one of the first video memory region and the second video memory region, the current size of at least one of the first video memory region and the second video memory region is changed. The global video memory manager is also configured to: in response to the current input parameter being different from the previous input parameter, release the current second video memory region and convert the current second video memory region into the empty video memory region to obtain a new empty video memory region.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 1-8.

11. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 1-8.

Images