Method and equipment for managing video memory and storage medium

Abstract

The embodiment of the invention relates to a method for managing a video memory. The method comprises the following steps: determining a global video memory area about a board card at the board card; determining a first video memory area for key value caching, and distributing the first video memory area to a first end of the global video memory area; determining a second video memory area used for activating value cache, and distributing the second video memory area to a second end of the global video memory area; and based on the first video memory area and the second video memory area, determining an unoccupied empty video memory area in the global video memory area. At least part of at least one of a first video memory area and a second video memory area is released to an empty video memory area, or at least part of the empty video memory area is converted into at least one of the first video memory area and the second video memory area; the current size of at least one of the first video memory area and the second video memory area can be dynamically changed, so that the first video memory area and the second video memory area are flexibly configured in the global video memory area.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of computer technology, and more particularly to a method, device, and storage medium for managing video memory. Background Art

[0002] Large language models (LLMs) generally use a Transformer structure to perform reasoning in a manner such as autoregressive reasoning. Usually, in order to reduce the amount of repeated calculations during the reasoning process, the previous calculation results (for example, keys and values, intermediate activation values, etc.) can be stored in a cache in advance to reduce the amount of calculations. However, in the prior art, the size of the pre-configured memory areas in the video memory, such as the memory area for the key value cache and the memory area for the activation value cache, is fixed and cannot be adjusted dynamically, and multiple memory areas are managed separately.

[0003] In summary, the existing solutions for managing video memory have the following shortcomings: the size of the video memory area for key value caching and the size of the video memory area for activation value caching in the video memory cannot be adjusted in a coordinated manner, resulting in low video memory utilization. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and system for managing video memory, so as to coordinately adjust the size of the video memory area used for key-value cache and the size of the video memory area used for activation value cache in the video memory, thereby improving video memory utilization and throughput.

[0005] According to a first aspect of the present invention, there is provided a method for managing video memory, comprising: determining, at a video card, a global video memory area regarding the video card; determining a first video memory area for a key-value cache, and allocating the first video memory area to a first end of the global video memory area; determining a second video memory area for an activation value cache, and allocating the second video memory area to a second end of the global video memory area; determining an unoccupied empty video memory area in the global video memory area based on the first video memory area and the second video memory area; and changing a current size of at least one of the first video memory area and the second video memory area by releasing at least a portion of at least one of the first video memory area and the second video memory area to the empty video memory area, or converting at least a portion of the empty video memory area to at least one of the first video memory area and the second video memory area, so as to achieve flexible configuration of the global video memory area.

[0006] In some embodiments, determining a global memory area for a board includes: determining a size of a weight memory area for model weight caching; determining a size of a spare memory area; and calculating a first predetermined size for the global memory area based at least on the size of the weight memory area and the size of the spare memory area.

[0007] In some embodiments, the first predetermined size is equal to the total video memory size of the board minus the size of the weighted video memory area and the size of the spare video memory area.

[0008] In some embodiments, determining a first video memory region for key-value caching includes: determining model parameters based on a model for inference; calculating a video memory block size based on the model parameters; and dividing the first video memory region into blocks based on the video memory block size.

[0009] In some embodiments, determining a second video memory area for caching activation values ​​includes: determining input parameters related to the input based at least on the input for inference, the input parameters including: batch size and sequence length; determining all activation tensors used for inference based on the input parameters, and determining the size of the video memory occupied by each activation tensor; and calculating the size of the video memory occupied by each activation tensor based on predetermined rules to obtain the size of the second video memory area.

[0010] In some embodiments, the first video memory area has a first video memory type, and the second video memory area has a second video memory type, wherein the first video memory type and the second video memory type are different.

[0011] According to a second aspect of the present invention, there is provided a method for accessing video memory managed according to the method of the first aspect of the present invention, comprising: determining whether an input parameter currently input is the same as an input parameter previously input; in response to the input parameter currently input being the same as the input parameter previously input, determining whether a first video memory area satisfies a threshold condition; in response to the first video memory area satisfying the threshold condition, accessing the video memory based on the current configuration of the global video memory area.

[0012] In some embodiments, the method of the second aspect of the present invention also includes: in response to the input parameters of the current input being different from the input parameters of the previous input, releasing the current second video memory area, converting the current second video memory area into an empty video memory area to obtain a new empty video memory area; based on the input parameters of the current input, determining the size of the new second video memory area; comparing the size of the new second video memory area with the size of the new empty video memory area; and in response to the size of the new second video memory area being less than or equal to the size of the new empty video memory area, allocating the new second video memory area to the second end of the global video memory area.

[0013] In some embodiments, the method of the second aspect of the present invention also includes: in response to the size of the new second video memory area being larger than the size of the new empty video memory area, determining whether the first video memory area includes an unused area; in response to the first video memory area including an unused area, releasing unused video memory blocks in the unused area to convert the video memory areas corresponding to the unused video memory blocks into empty video memory areas to obtain an expanded new empty video memory area; comparing the size of the new second video memory area with the size of the expanded new empty video memory area; and in response to the size of the new second video memory area being less than or equal to the expanded new empty video memory area, allocating the new second video memory area to the second end of the global video memory area.

[0014] According to a third aspect of the present invention, there is provided a device for managing video memory, comprising: a first video memory manager configured to manage a first video memory area for a key-value cache; a second video memory manager configured to manage a second video memory area for an activation value cache; and a global video memory manager configured to: determine, at a video card, a global video memory area for the video card; allocate a first video memory area in the global video memory area; and allocate a second video memory area in the global video memory area, wherein the current size of at least one of the first video memory area and the second video memory area is changed by releasing at least a portion of at least one of the first video memory area and the second video memory area to an unoccupied empty video memory area in the global video memory area, or converting at least a portion of the empty video memory area to at least one of the first video memory area and the second video memory area.

[0015] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the first aspect and the second aspect of the present invention.

[0016] According to a fifth aspect of the present invention, there is provided a computer program product, wherein the computer program product is tangibly stored on a non-transitory computer readable medium and comprises machine executable instructions which, when executed, cause a machine to perform the steps in the method of the first and second aspects of the present invention.

[0017] Compared with the existing solutions for managing video memory, the solution for managing video memory provided by the present invention can dynamically change the current size of at least one of the first video memory area for key-value caching and the second video memory area for activation value caching, thereby realizing flexible configuration of the first video memory area for key-value caching and the second video memory area for activation value caching in the global video memory area, thereby improving video memory utilization and throughput.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0020] Figure 1 A schematic diagram of a device for managing video memory according to an embodiment of the present invention is shown.

[0021] Figure 2 A flow chart of a method for managing video memory according to an embodiment of the present invention is shown.

[0022] Figure 3 A schematic diagram of a global video memory area according to an embodiment of the present invention is shown.

[0023] Figure 4 A flow chart of a method for accessing a video memory according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] Large language models (LLMs) generally use a Transformer structure to perform reasoning in a manner such as autoregressive reasoning. In general, in order to reduce the amount of repeated calculations during the reasoning process, KVCache (key-value cache) technology can be used to store previous calculation results (such as keys and values) in the cache in advance, so that the subsequent reasoning process can directly read these calculation results from the cache without repeated calculations, thereby significantly reducing the amount of calculations and improving reasoning efficiency. In this case, it is necessary to pre-apply for a whole block of storage space in the video memory such as the GPU for key-value caching. In order to better manage the storage space for key-value caching in the video memory, the PageAttention (paged attention mechanism) technology can be further used to store continuous key values ​​in discontinuous storage space, so that the video memory can be fully utilized, the video memory utilization and throughput can be improved, and the video memory waste can be avoided.

[0027] During the inference process of the large language model, the intermediate activation values ​​calculated during the forward pass will be stored in the video memory. Optionally, the key value can be cached to reduce repeated calculations and improve inference efficiency. Generally speaking, the size of the video memory occupied by storing intermediate activation values ​​is related to the input data size of the model. For example, the longer the sequence length of the model input, the larger the video memory occupied by storing intermediate activation values. In other words, when the application scenario of the large language model includes a large amount of long text, it is necessary to preset a larger storage space in the video memory for activation value caching.

[0028] In the above case, since the preset storage space for activation value caching is relatively large, the storage space that can be set in the video memory for key value caching is limited to a certain extent. However, the scheme for managing video memory, including the above-mentioned PageAttention technology, can only optimize the video memory area for key value caching, but cannot take into account the video memory area for activation value caching. Therefore, the prior art lacks a video memory optimization scheme that can coordinately adjust the video memory area for key value caching and the video memory area for activation value caching in the video memory.

[0029] Furthermore, in application scenarios where long and short texts are mixed, especially application scenarios where there is a small amount of long text and a large amount of short text, since the size of the storage space preset in the video memory for the activation value cache needs to take into account the size of the video memory occupied by the long text activation, when reasoning is performed on the short text, this part of the video memory occupied for the activation value cache will be obviously wasted, resulting in low video memory utilization and a decrease in the overall video memory throughput.

[0030] In summary, the shortcomings of existing solutions for managing video memory are that it is impossible to coordinately adjust the size of the video memory area used for key-value caching and the size of the video memory area used for activation value caching, and in application scenarios with mixed long and short texts, the video memory utilization is low.

[0031] In order to at least partially solve one or more of the above problems and other potential problems, an exemplary embodiment of the present invention proposes a solution for managing video memory. The solution includes: at a board, determining a global video memory area for the board; determining a first video memory area for key-value caching, and allocating the first video memory area to a first end of the global video memory area; determining a second video memory area for activation value caching, and allocating the second video memory area to a second end of the global video memory area; based on the first video memory area and the second video memory area, determining an unoccupied empty video memory area in the global video memory area. By releasing at least part of at least one of the first video memory area and the second video memory area to an empty video memory area, or converting at least part of the empty video memory area to at least one of the first video memory area and the second video memory area, the current size of at least one of the first video memory area and the second video memory area can be dynamically changed, thereby realizing flexible configuration of the first video memory area for key-value caching and the second video memory area for activation value caching in the global video memory area.

[0032] The following will be combined Figures 1 to 3 A scheme for managing video memory according to an embodiment of the present invention is described in detail.

[0033] Figure 1 A schematic diagram of a device 100 for managing video memory according to an embodiment of the present invention is shown. It should be understood that the device 100 may also include additional devices not shown and / or may integrate one or more of the devices shown into one or more devices, and the scope of the present invention is not limited in this respect.

[0034] like Figure 1 As shown, the device 100 may include: a first graphics memory manager 101 , a second graphics memory manager 102 , and a global graphics memory manager 103 .

[0035] Regarding the first video memory manager 101, it can be configured to manage the first video memory area for the key value cache (KVCache). According to an embodiment of the present invention, the first video memory manager 101 can be configured to, for example, apply to the global video memory manager 103 for additional storage space for the key value cache. In some other embodiments, the first video memory manager 101 can also be configured to release unused storage space to the global video memory manager 103.

[0036] The second video memory manager 102 may be configured to manage the second video memory region for activating value cache. According to an embodiment of the present invention, the second video memory manager 102 may be configured to, for example, calculate the size of the second video memory region for activating value cache.

[0037] The global video memory manager 103 may be configured to: determine the global video memory area of ​​the board at the board; allocate the first video memory area in the global video memory area; allocate the second video memory area in the global video memory area. The "board" here may include but is not limited to: a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), a neural network processing unit (NPU), etc.

[0038] According to an embodiment of the present invention, the device 100 may be further configured to: release at least part of at least one of the first video memory area and the second video memory area to an empty video memory area, or convert at least part of the empty video memory area into at least one of the first video memory area and the second video memory area, via the first video memory manager 101, the second video memory manager 102, and the global video memory manager 103, so as to change the current size of at least one of the first video memory area and the second video memory area, so as to dynamically manage the first video memory area and the second video memory area, and realize flexible configuration of the global video memory area. Figure 2 Detailed Description Figure 1 How does the device 100 implement flexible configuration of the global video memory area?

[0039] Figure 2 The flowchart of the method 200 for managing video memory according to an embodiment of the present invention is shown. It should be understood that the method 200 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0040] In step 201 , at the board, the global video memory manager 103 determines the global video memory area of ​​the board.

[0041] Regarding the global video memory area, according to an embodiment of the present invention, it may refer to a whole block of storage space pre-divided in the video memory of the board for subsequently storing key values ​​and activation values ​​therein. Specifically, the global video memory area may include: a first video memory area for key value caching, a second video memory area for activation value caching, and an empty video memory area, wherein the empty video memory area may be configured to provide additional storage space to the first video memory area and / or the second video memory area.

[0042] Typically, in the inference process of a large language model, in addition to key values ​​and activation values, data such as model weights are also required to be stored in the video memory of the board. Therefore, in some embodiments of the present invention, the storage space and size of the global video memory area in the video memory of the board can be determined based at least on the storage space used to store the model weights.

[0043] Specifically, according to an embodiment of the present invention, determining the global video memory area regarding the board may further include: determining the size of a weight video memory area for model weight caching; determining the size of a spare video memory area; and calculating a first predetermined size for the global video memory area based at least on the size of the weight video memory area and the size of the spare video memory area.

[0044] Regarding the weight memory area used for model weight caching, as described above, it may refer to a storage space used to store model weights.

[0045] The spare video memory area may refer to the size of the storage space in the video memory as a buffer. It should be understood that in practice, a small amount of spare storage space is usually reserved in the video memory of the board as a buffer to buffer input and / or output data.

[0046] According to an embodiment of the present invention, other video memory areas in the video memory of the board except the weighted video memory area and the spare video memory area can be applied as global video memory areas for storing key values ​​and activation values. It can be seen that in an embodiment of the present invention, the size of the storage space that can be used as the global video memory area in the video memory of the board can be calculated by subtracting the size of the weighted video memory area from the total video memory size of the board, and then subtracting the size of the spare video memory area. In other words, the first predetermined size of the global video memory area can be equal to the total video memory size of the board minus the size of the weighted video memory area and the size of the spare video memory area.

[0047] As can be seen from the above, in the solution for managing video memory provided by the present invention, the global video memory manager 103 can determine a whole storage space with a first predetermined size in the video memory of the board, such as based on a calculated first predetermined size for the global video memory area, to serve as the global video memory area of ​​the board.

[0048] According to the inventive concept of the present invention, the global video memory manager 103 may be initialized based on the determined global video memory area with the first predetermined size. Specifically, the global video memory manager 103 may record the initial address and size of the global video memory area.

[0049] In step 202 , the global video memory manager 103 determines a first video memory region for key-value caching, and allocates the first video memory region to a first end of the global video memory region.

[0050] According to the inventive concept of the present invention, the size and position of the first video memory area may be determined by the global video memory manager 103 , wherein the first video memory area is configured to store key values.

[0051] by Figure 3 Taking the global video memory area 300 shown in FIG. Figure 3 As shown, the global memory manager 103 may determine a first memory region 301 for key-value caching and allocate the first memory region 301 to one end of the global memory region 300 , such as an initial position of the global memory region 300 .

[0052] The size of the first video memory area 301 for key-value caching may be determined based on empirical values. In some embodiments, the size of the first video memory area 301 for key-value caching may be determined by inputting data of a specified batch size into the model in advance and then performing calculations.

[0053] According to the inventive concept of the present invention, the first video memory manager 101 can then divide the first video memory area 301 into blocks to obtain multiple video memory blocks, so that the first video memory area 301 can be managed based on the video memory blocks (for example, applying for additional storage space, releasing excess storage space, etc.).

[0054] Regarding the video memory block, it may refer to a video memory sub-area with the same storage space. According to an embodiment of the present invention, the first video memory manager 101 may first determine the size of the video memory block, and then divide the first video memory area 301 into blocks based on the determined size of the video memory block. Specifically, in an embodiment of the present invention, the first video memory manager 101 may further determine model parameters based on the model used for inference; calculate the size of the video memory block based on the model parameters; and divide the first video memory area 301 into blocks based on the size of the video memory block.

[0055] Regarding model parameters, according to an embodiment of the present invention, they may include: the number of heads (num_heads) of the model's attention mechanism and the size of each head (head_size), the number of layers (num_layers) of the model, and the size corresponding to the accuracy of the model (datatype_size).

[0056] Assuming that each video memory block can store n (n is an integer greater than 0) token key-value caches, and recorded as "slot_size", according to some embodiments of the present invention, combined with the multiple model parameters listed above, the size (block_size) of each video memory block can be calculated according to the following formula (1).

[0057] (1) Then, based on the calculated size (block_size) of each video memory block, the first video memory manager 101 can divide the first video memory area 301 into blocks to obtain N video memory blocks of the same size. Figure 3 As shown, N is equal to the size of the first video memory area 301 divided by the size of each video memory block (block_size).

[0058] In step 203 , the global video memory manager 103 determines a second video memory area for activating the value cache, and allocates the second video memory area to the second end of the global video memory area.

[0059] According to the inventive concept of the present invention, the size and position of the second video memory area may be determined by the global video memory manager 103 , wherein the second video memory area is configured to store the activation value.

[0060] Still Figure 3 Taking the global video memory area 300 shown in FIG. 1 as an example, the global video memory manager 103 may determine a second video memory area 302 for activating the value cache, and allocate the second video memory area 302 to the other end of the global video memory area 300, such as Figure 3 At the end of the global video memory area 300.

[0061] Regarding the size of the second memory area 302 for activating the value cache, it may be related to the total number of tokens involved in model reasoning, that is, to the batch size and sequence length of the data to be input into the model. Generally, the larger the batch size, the larger the size of the second memory area 302 for activating the value cache; the larger the sequence length, the larger the size of the second memory area 302 for activating the value cache.

[0062] According to the inventive concept of the present invention, the size of the second video memory area 302 determined for activating the value cache may vary with the change of the input input into the model for inference. For example, in an embodiment of the present invention, determining the second video memory area 302 for activating the value cache may include: the second video memory manager 102 determines input parameters related to the input based at least on the input for inference; determining all activation tensors for inference according to the input parameters, and determining the video memory size occupied by each activation tensor; and calculating the video memory size occupied by each activation tensor based on a predetermined rule to obtain the size of the second video memory area 302.

[0063] Regarding input parameters, these may include things like batch size and sequence length.

[0064] The predetermined rule may refer to a summation rule or a reuse rule, etc. According to some embodiments of the present invention, the size of the video memory occupied by each activated tensor may be summed to obtain the size of the second video memory area 302. According to some other embodiments of the present invention, the size of the video memory occupied by each activated tensor may be calculated based on the reuse rule, that is, based on the life cycle of each activated tensor, the size of the video memory occupied by each activated tensor is calculated according to the reuse rule of the video memory to obtain the size of the second video memory area 302.

[0065] Regarding the reuse rule, according to an embodiment of the present invention, it may be, for example, a video memory reuse rule based on an implicit free list (Implicit Free List).

[0066] Taking the calculation of a large language model reasoning embodied as a graph as an example, the output data and temporary data of each node in the graph to be calculated can be regarded as activation values. As the input parameters change, that is, as the batch size and sequence length corresponding to the current input change, the size of the output data and temporary data of each node also changes accordingly. In view of this feature, the inventive concept of the present invention also includes: whenever the input changes, that is, the batch size or sequence length of the input changes, the size of the activation value corresponding to each node is recalculated, and on this basis, the size of the second video memory area used for caching activation values ​​is re-determined based on the reuse rule.

[0067] According to an embodiment of the present invention, the second video memory manager 102 can determine the video memory size occupied by all activation tensors and each activation tensor used for reasoning based at least on the batch size and sequence length of the input for reasoning, wherein the video memory size occupied by each activation tensor is not necessarily the same. Then, the second video memory manager 102 can calculate the video memory size occupied by each activation tensor based on a predetermined rule (for example, summation or calculation based on the reuse rule of the implicit free list) to calculate the total video memory size occupied by all activation tensors, thereby obtaining the size of the second video memory area 302. On this basis, the global video memory manager 103 can allocate the second video memory area 302 to the end of the global video memory area 300 as described above based on the obtained size of the second video memory area 302.

[0068] Furthermore, in some embodiments, due to the use of reuse rules to determine the size of the second video memory area used for activation value caching, when a node in a graph of large language model inference is calculated and the data calculated by the node is not used by other nodes in the graph, the video memory occupied by the activation value of the node can be released and used by the activation values ​​of other nodes. As a result, the size of the video memory occupied by the second video memory area used for activation value caching can be further reduced, so that more video memory space can be used for key value caching.

[0069] In addition, according to some other embodiments of the present invention, the determination of the size of the second video memory area for activating the value cache and the allocation of the second video memory area in the global video memory area only occur when the input parameters change, that is, when the pre-filling and decoding stages of inference are switched, thereby significantly reducing the time for calculating the size of the second video memory area for activating the value cache and allocating the second video memory area.

[0070] In step 204 , the global video memory manager 103 determines an unoccupied empty video memory area in the global video memory area based on the first video memory area and the second video memory area.

[0071] According to the inventive concept of the present invention, an empty video memory area can be configured in the global video memory area to realize flexible configuration of the first video memory area and the second video memory area in the global video memory area. The empty video memory area can refer to a storage space in the global video memory area that is neither allocated for storing key values ​​nor allocated for storing activation values.

[0072] According to an embodiment of the present invention, the global video memory manager 103 may determine the area of ​​the global video memory area other than the first video memory area and the second video memory area, ie, the unoccupied part of the global video memory area, as an empty video memory area.

[0073] by Figure 3 The global video memory area 300 shown in FIG. Figure 3 As shown, the area other than the first video memory area 301 and the second video memory area 302 in the global video memory area 300 , that is, the video memory area between the first video memory area 301 and the second video memory area 302 , can be regarded as an empty video memory area 303 .

[0074] In order to take into account the size of the video memory area for key-value caching and the size of the video memory area for activation value caching configured in the global video memory area, and to achieve dynamic adjustment of the two, according to the inventive concept of the present invention, while keeping the size of the global video memory area 300 unchanged, the relative sizes of the first video memory area 301, the second video memory area 302 and the empty video memory area 303 can be changed to achieve flexible configuration of the global video memory area 300.

[0075] Specifically, in step 205, by releasing at least a portion of at least one of the first video memory area 301 and the second video memory area 302 to the empty video memory area 303, or converting at least a portion of the empty video memory area 303 to at least one of the first video memory area 301 and the second video memory area 302, the current size of at least one of the first video memory area 301 and the second video memory area 302 is changed to achieve flexible configuration of the global video memory area 300.

[0076] Regarding releasing at least part of at least one of the first video memory area 301 and the second video memory area 302 to the empty video memory area 303, it may refer to: releasing at least part of the first video memory area 301 to the empty video memory area 303 to reduce the size of the first video memory area 301 and increase the size of the empty video memory area 303, while keeping the size of the second video memory area 302 unchanged; or, releasing the entire second video memory area 302 to the empty video memory area 303 to reduce the size of the second video memory area 302 and increase the size of the empty video memory area 303, while keeping the size of the first video memory area 301 unchanged.

[0077] According to some embodiments of the present invention, the first video memory manager 101 may release at least a portion of the first video memory area 301 adjacent to the empty video memory area 303 to the empty video memory area 303. According to still other embodiments of the present invention, the second video memory manager 102 may release the entire second video memory area 302 to the empty video memory area 303.

[0078] Regarding converting at least part of the empty video memory area 303 into at least one of the first video memory area 301 and the second video memory area 302, it may refer to: converting at least part of the empty video memory area 303 into the first video memory area 301 to increase the size of the first video memory area 301 and reduce the size of the empty video memory area 303 while keeping the second video memory area 302 unchanged; or, converting at least part of the empty video memory area 303 into the second video memory area 302 while keeping the size of the first video memory area 301 unchanged.

[0079] According to some embodiments of the present invention, the first video memory manager 101 may send a request for applying for additional storage space to the global video memory manager 103, so that the global video memory manager 103 may convert at least a portion of the empty video memory area 303 adjacent to the first video memory area 301 into the first video memory area 301. According to still other embodiments of the present invention, the second video memory manager 102 may determine the size of the empty video memory area 303 to be converted into the second video memory area 302, and the global video memory manager 103 may convert the area at the end of the empty video memory area 303 into the second video memory area 302 based on the determined size of the empty video memory area 303 to be converted into the second video memory area 302.

[0080] It can be seen from the above that the method for managing video memory provided in accordance with an embodiment of the present invention can take into account the video memory area used for key-value caching and the video memory area used for activation value caching in the video memory, and dynamically adjust the relative sizes of the video memory area used for key-value caching and the video memory area used for activation value caching with the help of empty video memory areas, so as to improve video memory utilization and throughput.

[0081] Further, it can be seen from the above that the device for managing video memory provided by the present invention can implement two-level management of the video memory at the board, that is, the first video memory manager of the device (such as Figure 1 A first video memory manager 101 and a second video memory manager (such as Figure 1 The second video memory manager 102) respectively implements the first video memory area (such as Figure 3 A first video memory area 301 and a second video memory area (such as Figure 3 The second video memory area 302) is managed by a global video memory manager (such as Figure 1 The global video memory manager 103) implements the global video memory area (such as Figure 3 The global video memory area 300) enables more flexible management of the first video memory area and the second video memory area, improves video memory utilization, and reduces the number of times of applying for storage space and / or releasing storage space involved in adjusting the size of the first video memory area and the second video memory area, thereby improving efficiency.

[0082] Moreover, in the above technical solution provided by the embodiment of the present invention, the first video memory area and the second video memory area are respectively managed by different video memory managers, that is, the video memory area for key-value caching and the video memory area for activation value caching are managed separately, so that different management methods can be used for the video memory area for key-value caching and the video memory area for activation value caching according to the actual usage scenario. For example, in some embodiments of the present invention, the first video memory area can be managed using the first management method, and the second video memory area can be managed using the second management method. In one example, the first video memory area can be managed using a paging attention mechanism, and the second video memory area can be managed using a dedicated method different from the paging attention mechanism. Therefore, the scheme for managing video memory proposed in the present invention can also realize the management of video memory areas for key-value caching and video memory areas for activation value caching with different video memory types.

[0083] According to an embodiment of the present invention, a first video memory area (such as Figure 3 The first video memory area 301 has a first video memory type, and the second video memory area (such as Figure 3 The second memory region 302 of the key cache has a second memory type different from the first memory type. For example, the first memory type of the first memory region for the key-value cache may be a NUMA (non-uniform memory access) memory type, and the second memory type of the second memory region for the activation value cache may be a UMA (uniform memory access) memory type.

[0084] The following will be combined Figure 4A detailed description is given of how to access the video memory managed by the method for managing video memory provided according to an embodiment of the present invention.

[0085] Figure 4 FIG. 4 is a flowchart of a method 400 for accessing video memory according to an embodiment of the present invention. According to an embodiment of the present invention, the method 400 may be performed by a processor such as Figure 1 It should be understood that the method 400 may further include additional actions not shown and / or may omit the actions shown, and the scope of the present invention is not limited in this respect.

[0086] In step 401, at a board, a global video memory area of ​​the board is determined.

[0087] According to an embodiment of the present invention, the global memory manager 103 may determine the global memory area of ​​the board at the board. For example, the global memory manager 103 may determine all memory areas of the board except for those used to store weights and as spares as the global memory area of ​​the board.

[0088] Regarding step 401, please refer to the above description of step 201 for details, which will not be repeated here.

[0089] In step 402, a first video memory region for key-value caching is determined, and the first video memory region is allocated to a first end of a global video memory region.

[0090] According to an embodiment of the present invention, the global memory manager 103 may determine a first memory region for key-value caching, and allocate the first memory region to a first end of the global memory region. For example, the global memory manager 103 may determine a first memory region for key-value caching, and allocate the first memory region to an initial position of the global memory region.

[0091] Regarding step 402, please refer to the above description of step 202 for details, which will not be repeated here.

[0092] In step 403, a second video memory area for activating the value cache is determined, and the second video memory area is allocated to a second end of the global video memory area.

[0093] According to an embodiment of the present invention, the global video memory manager 103 may determine the second video memory area for activating the value cache, and allocate the second video memory area to the second end of the global video memory area. For example, the global video memory manager 103 may determine the second video memory area for activating the value cache, and allocate the second video memory area to the end of the global video memory area.

[0094] Regarding step 403, please refer to the above description of step 203 for details, which will not be repeated here.

[0095] In step 404, based on the first video memory area and the second video memory area, an unoccupied empty video memory area in the global video memory area is determined.

[0096] According to an embodiment of the present invention, the global video memory manager 103 may determine an unoccupied empty video memory region in the global video memory region based on the first video memory region and the second video memory region. For example, the global video memory manager 103 may determine an unoccupied global video memory region between the first video memory region and the second video memory region as an empty video memory region.

[0097] Regarding step 404, its description can refer to the description of step 204 above, and will not be repeated here.

[0098] Based on the above steps 401 to 404, the initial configuration of the global video memory area can be completed at the board. It should be understood that according to the inventive concept of the present invention, the initial configuration can be dynamically changed in the subsequent model reasoning process.

[0099] In step 405, it is determined whether the input parameters currently input are the same as the input parameters previously input.

[0100] According to the inventive concept of the present invention, since the size of the second video memory area required for large language model reasoning is related to the input of the reasoning, when performing each reasoning, it is possible to determine whether the size of the required second video memory area has changed by judging whether the input used for reasoning has changed. For example, it is possible to determine whether the shape of the current input has changed compared to the previous input by judging whether the shape of the input used for reasoning has changed. According to some embodiments of the present invention, it is possible to determine whether the shape of the input used for reasoning has changed by comparing the input parameters of the current input with the input parameters of the previous input.

[0101] If the input parameters of the current input are the same as the input parameters of the previous input, it means that the shape of the input used for reasoning has not changed. Therefore, it can be known that the size of the second video memory area corresponding to the current reasoning is the same as the size of the second video memory area corresponding to the previous reasoning, that is, the current reasoning can be performed on the basis of the second video memory area determined for the previous reasoning. In this case, it can be further determined whether the size of the first video memory area meets the requirements for performing the current reasoning to determine whether reasoning can also be performed on the basis of the current first video memory area. Therefore, according to an embodiment of the present invention, in response to determining at step 405 that the input parameters of the current input are the same as the input parameters of the previous input, the process proceeds to step 406 to determine whether the first video memory area meets the threshold condition.

[0102] The threshold condition may refer to a condition that indicates that the first memory region has sufficient storage space. According to an embodiment of the present invention, the threshold condition may include, for example, that the currently available storage space of the first memory region is sufficient to store the currently inferred key value.

[0103] According to an embodiment of the present invention, the first memory manager 101 may determine whether the first memory region meets the threshold condition. For example, the first memory manager 101 may determine whether the currently available storage space of the first memory region is sufficient to store the currently inferred key value.

[0104] If the first video memory area meets the threshold condition, that is, the current available storage space of the first video memory area is sufficient to store the key value of the current inference, it means that the current inference can be completed based on the first video memory area and the second video memory area configured in the current global video memory area. Therefore, in response to determining that the first video memory area meets the threshold condition at step 406, the process proceeds to step 407 to access the video memory based on the current configuration of the global video memory area.

[0105] If the first video memory area does not meet the threshold condition, that is, the current available storage space of the first video memory area is not enough to store the key value of the current reasoning, in this case, it is necessary to convert at least part of the empty video memory area into the first video memory area, and expand the available storage space of the first video memory area so that the current reasoning can be completed based on the expanded available storage space of the first video memory area. Therefore, according to an embodiment of the present invention, in response to determining that the first video memory area does not meet the threshold condition at step 406, the process proceeds to step 408 to convert at least part of the empty video memory area into the first video memory area.

[0106] According to an embodiment of the present invention, the global video memory manager 103 may convert at least part of the empty video memory area into the first video memory area. For example, in some embodiments, the first video memory manager 101 sends a request for applying for additional storage space to the global video memory manager 103 at the granularity of the video memory block size, wherein the request for applying for additional storage space sent may include information indicating the size of the additional storage space to be applied for (e.g., information related to the number of video memory blocks, etc.). In response to receiving this request, the global video memory manager 103 converts at least part of the empty video memory area adjacent to the first video memory area (even all the empty video memory areas) into the first video memory area, thereby obtaining an enlarged first video memory area. As a result, the available storage space of the enlarged first video memory area at least includes the part of the first video memory area converted from the empty video memory area, so that the available storage space of the enlarged first video memory area can meet the needs of completing the current reasoning. On this basis, the process may then proceed to step 407, accessing the video memory based on the current configuration of the global video memory area, wherein the current configuration of the global video memory area includes the enlarged first video memory area.

[0107] In some embodiments, if the available storage space of the expanded first video memory area obtained by converting all empty video memory areas into the first video memory area still cannot meet the needs of completing the current reasoning, given that the current global video memory area is already completely occupied by the first video memory area and the second video memory area, a swap operation can be performed, for example, to swap the key values ​​of the lower priority statements stored outside the current board to other boards (such as the central processing unit (CPU)) so as to free up some storage space in the global video memory area of ​​the current board for storing the key values ​​of higher priority statements.

[0108] Returning to step 405, if the input parameters of the current input are different from the input parameters of the previous input, it means that the shape of the input for reasoning has changed. According to the inventive concept of the present invention, it is necessary to re-determine the size of the second video memory area for activating the value cache. Therefore, according to an embodiment of the present invention, in response to determining at step 405 that the input parameters of the current input are different from the input parameters of the previous input, proceed to step 409, release the current second video memory area, determine a new second video memory area, and allocate the new second video memory area to the second end of the global video memory area.

[0109] According to an embodiment of the present invention, the global video memory manager 103 may release the current second video memory area, determine a new second video memory area, and allocate the new second video memory area to the second end of the global video memory area. For example, the global video memory manager 103 may release the current second video memory area to convert all current second video memory areas into empty video memory areas; based on the currently inputted input parameters, the global video memory manager 103 re-determines the second video memory area (which may be recorded as "new second video memory area") for activating the value cache, and allocates the new second video memory area to the second end (i.e., the end position) of the global video memory area, that is, the global video memory manager 103 allocates the new second video memory area to the empty video memory area and away from the first video memory area. For the specific steps of determining the new second video memory area for activating the value cache, please refer to the description of Figure 2 The description of step 203 is not repeated here.

[0110] However, in some embodiments, before allocating the new second video memory area to the second end of the global video memory area, in order to ensure that there is enough storage space in the global video memory area, the video memory size required to be occupied by the new second video memory area may be compared with the available storage space in the global video memory area except for the configured first video memory area. Therefore, for these embodiments, the above step 409 may further include: releasing the current second video memory area to convert the current second video memory area into an empty video memory area to obtain a new empty video memory area; determining the size of the new second video memory area based on the currently input input parameters; comparing the size of the new second video memory area with the size of the new empty video memory area; and in response to the size of the new second video memory area being less than or equal to the size of the new empty video memory area, allocating the new second video memory area to the second end of the global video memory area.

[0111] For example, the global video memory manager 103 releases the current second video memory area to convert the current second video memory area into an empty video memory area, thereby obtaining a new empty video memory area. At this time, the global video memory area only includes the first video memory area and the new empty video memory area. On this basis, the second video memory manager 102 can determine the size of the new second video memory area based on the input parameters of the current input (including the batch size and sequence length of the current input), for example, by re-determining the video memory size occupied by all activation tensors used for inference and each activation tensor through the input parameters of the current input. Then, the global video memory manager 103 can compare the size of the new second video memory area with the size of the new empty video memory area to determine whether the size of the new second video memory area is less than or equal to the size of the new empty video memory area.

[0112] In some embodiments, if the size of the new second video memory area is less than or equal to the size of the new empty video memory area, it means that there is sufficient storage space in the current global video memory area for configuring the new second video memory area. The global video memory manager 103 can allocate the new second video memory area to the second end of the global video memory area to achieve reconfiguration of the second video memory area of ​​the global video memory area.

[0113] In some other embodiments, if the size of the new second video memory area is larger than the size of the new empty video memory area, it means that there is not enough storage space in the current global video memory area for configuring the new second video memory area. In this case, it is possible to optionally release the unused storage space in the first video memory area so that there is enough storage space in the global video memory area to configure the new second video memory area. Specifically, for these embodiments, it is further determined whether the first video memory area includes an unused area; in response to the first video memory area including the unused area, unused video memory blocks in the unused area are released to convert the video memory area corresponding to the unused video memory blocks into an empty video memory area to obtain an expanded new empty video memory area; the size of the new second video memory area is compared with the size of the expanded new empty video memory area; and in response to the size of the new second video memory area being less than or equal to the expanded new empty video memory area, the new second video memory area is allocated to the second end of the global video memory area.

[0114] For example, the first video memory manager 101 may determine whether the first video memory area includes an unused area; in response to the first video memory area including the unused area, the global video memory manager 103 releases the unused video memory blocks in the unused area of ​​the first video memory area to convert the video memory area corresponding to the unused video memory blocks in the first video memory area into an empty video memory area, and obtain an expanded new empty video memory area; the global video memory manager 103 compares the size of the new second video memory area with the size of the expanded new empty video memory area; and in response to the size of the new second video memory area being less than or equal to the expanded new empty video memory area, the global video memory manager 103 allocates the new second video memory area to the second end of the global video memory area.

[0115] Regarding determining whether the first video memory area includes an unused area, the first video memory manager 101 may determine whether the end of the first video memory area (ie, the position adjacent to the first video memory area and the empty video memory area) includes an unused video memory block.

[0116] Regarding releasing the unused video memory blocks in the unused area, the first video memory manager 101 may release part or all of the unused video memory blocks in the unused area.

[0117] However, in some embodiments, if the first video memory area does not include an unused area, or the size of the new second video memory area is larger than the expanded new empty video memory area, that is, the global video memory area is currently unable to accommodate the configuration of the new second video memory area therein, according to an embodiment of the present invention, the size of the second video memory area corresponding to the current reasoning can be reduced by, for example, reducing the input size used for the current reasoning (such as the number of scheduled statements), or the scheduling of the current statement can be postponed.

[0118] As can be seen from the above, according to the method for accessing video memory provided by the present invention, the global video memory area at the board can be flexibly configured based on the input for reasoning, so that the size of the first video memory area for key value cache and the second video memory area for activation value cache in the global video memory area can be adjusted in a coordinated manner, thereby improving the video memory utilization and overall throughput. In particular, when the input for model reasoning includes long text and short text, the solution provided by the present invention can significantly improve the video memory utilization and overall throughput.

[0119] Combination Figure 1 For device 100, an exemplary embodiment of a solution for managing and accessing video memory according to the present invention is given below as an example.

[0120] Step 1: In the inference initialization stage of the large language model, the device 100 determines all the memory areas of the board except for storing weights and as a backup as the global memory area of ​​the board, and initializes the global memory manager 103 with the determined global memory area. According to an embodiment of the present invention, the global memory manager 103 can be configured to manage the unoccupied storage space in the global memory area. For example, the global memory manager 103 can be configured to record the initial address of the global memory area, the offset and size of the unoccupied storage space in the global memory area. At this time, the value of the offset of the unoccupied storage space in the global memory area is 0, and the size of the unoccupied storage space in the global memory area is equal to the size of the determined entire global memory area.

[0121] Step 2: The global video memory manager 103 determines a first video memory area for key-value caching in the global video memory area determined in step 1, and the first video memory manager 101 divides the first video memory area into a plurality of video memory blocks of the same size. At this time, the value of the offset of the unoccupied storage space in the global video memory area is equal to the size of the first video memory area, and the size of the unoccupied storage space in the global video memory area is equal to the size of the entire global video memory area minus the size of the first video memory area.

[0122] Step 3: The second video memory manager 102 traverses the computation graph, such as by using memory reuse, to calculate the relative offset position and size of the storage space occupied by each activation tensor in the computation graph, and based on the storage space occupied by each activation tensor, calculates the size of the second video memory area used for caching activation values. The global video memory manager 103 can then allocate the second video memory area to the end of the global video memory area determined in step 1. At this time, the offset value of the unoccupied storage space in the global video memory area is equal to the size of the first video memory area, and the size of the unoccupied storage space in the global video memory area is equal to the size of the entire global video memory area minus the size of the first video memory area, minus the size of the second video memory area.

[0123] Step 4: For each inference, check whether the shape of the input used for the current inference (i.e., input parameters) has changed. If the shape of the input used for the current inference has changed, proceed to step 5; if the shape of the input used for the current inference has not changed, proceed to step 11.

[0124] Step 5: In response to the change in the shape of the input of the current inference, the global memory manager 103 releases the original second memory area, that is, converts the second memory area corresponding to the previous inference into an empty memory area. On this basis, the shape of the input of the current inference is used to recalculate the relative offset position and size of the storage space occupied by all activation tensors in the calculation graph, and the size of the new second memory area used for the activation value cache is calculated.

[0125] Step 6: The global video memory manager 103 determines whether the size of the unoccupied storage space in the global video memory area is greater than or equal to the size of the new second video memory area counted in step 5. If the size of the unoccupied storage space in the global video memory area is greater than or equal to the size of the new second video memory area, proceed to step 10; if the size of the unoccupied storage space in the global video memory area is less than the size of the new second video memory area, proceed to step 7.

[0126] Step seven: If the global video memory manager 103 determines that the size of the unoccupied storage space in the global video memory area is smaller than the size of the new second video memory area, the first video memory manager 101 releases the unused video memory block at the end of the first video memory area to the global video memory manager 103, that is, converts the unused video memory block at the end of the first video memory area into an empty video memory area to meet the requirement of the video memory size occupied by the new second video memory area.

[0127] Step 8: After the first video memory manager 101 releases the unused video memory block at the end of the first video memory area, it determines whether the unoccupied storage space in the global video memory area meets the requirements for configuring a new second video memory area. If the unoccupied storage space in the global video memory area can meet the requirements for configuring a new second video memory area, proceed to step 10; if the unoccupied storage space in the global video memory area cannot meet the requirements for configuring a new second video memory area, proceed to step 9.

[0128] Step 9: After the first video memory manager 101 releases the unused video memory block at the end of the first video memory area, the unoccupied storage space in the global video memory area still cannot meet the demand for configuring a new second video memory area, and the number of statements scheduled for this reasoning can be reduced to meet the demand for sufficient video memory. In some cases, if reducing to only one statement for this reasoning still cannot meet the demand for configuring a new second video memory area, the scheduling of the current statement can be postponed.

[0129] Step 10: The global video memory manager 103 allocates a new second video memory area at the end of the global video memory area, wherein the new second video memory area is managed by the second video memory manager 102. The physical address of the activated tensor is equal to the initial address of the global video memory area recorded in the global manager plus the offset of the unoccupied storage space in the global video memory area, plus the size of the unoccupied storage space in the global video memory area, and plus the relative offset of the activated tensor in the second video memory manager 102.

[0130] Step 11: The first video memory manager 101 determines whether the unused storage space in the first video memory area is sufficient. If there is sufficient unused storage space in the first video memory area, proceed to step 12; if there is not sufficient unused storage space in the first video memory area, proceed to step 15.

[0131] Step 12: The first video memory manager 101 applies for additional storage space from the global video memory manager 103 at the granularity of the video memory block size to expand the unused storage space in the first video memory area to meet the usage demand. Accordingly, the unoccupied storage space in the global video memory area managed by the global video memory manager 103 will be reduced.

[0132] Step 13: The first video memory manager 101 determines whether the additional storage space requested by the first video memory manager 101 meets the usage requirement. If the additional storage space requested by the first video memory manager 101 meets the usage requirement, proceed to step 15; if the additional storage space requested by the first video memory manager 101 does not meet the usage requirement, proceed to step 14.

[0133] Step 14: When the first video memory manager 101 applies for all the unoccupied storage space in the global video memory area and the usage demand is still not met, since the storage space of the entire global video memory area is fully occupied and there is no extra storage space available, in this case, a swap operation can be performed to swap the first video memory area occupied by the lower priority statements to a partition such as the CPU, so as to free up part of the storage space in the first video memory area for use by the higher priority statements.

[0134] Step 15: When the current reasoning is finished, check whether a new reasoning request is received. If a new reasoning request is received, return to step 4; if no new reasoning request is received, proceed to step 16.

[0135] Step 16: Check whether the termination condition is met to end the reasoning. If the reasoning is ended, proceed to step 17; if the reasoning continues, return to step 4.

[0136] Step 16: Deinitialize the first video memory manager 101, the second video memory manager 102 and the global video memory manager 103, and release the global video memory area at the board.

[0137] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for executing various aspects of the present invention. The computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0138] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the central processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0140] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0142] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0143] The above are only optional embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for managing video memory, characterized in that: include: At the board, determining a global video memory area for the board; Determine a first video memory area for key-value caching, and allocate the first video memory area to a first end of the global video memory area; Determine a second video memory area for activating the value cache, and allocate the second video memory area to a second end of the global video memory area; Determine an unoccupied empty video memory area in the global video memory area based on the first video memory area and the second video memory area; as well as By releasing at least a portion of at least one of the first video memory area and the second video memory area to the empty video memory area, or converting at least a portion of the empty video memory area to at least one of the first video memory area and the second video memory area, a current size of at least one of the first video memory area and the second video memory area is changed, so as to realize flexible configuration of the global video memory area, Wherein, determining the second video memory area for activating the value cache includes: Based at least on the input for inference, determining input parameters related to the input, the input parameters comprising: a batch size and a sequence length; According to the input parameters, determine all activation tensors used for reasoning, and determine the size of video memory occupied by each activation tensor; and The size of the video memory occupied by each activation tensor is calculated based on a predetermined rule to obtain the size of the second video memory area.

2. The method according to claim 1, characterized in that Determining the global video memory area of ​​the board includes: Determine the size of the weight memory area used for model weight caching; determining the size of the spare video memory region; and A first predetermined size for the global video memory area is calculated based at least on the size of the weighted video memory area and the size of the spare video memory area.

3. The method according to claim 2, characterized in that The first predetermined size is equal to the total video memory size of the board minus the size of the weighted video memory area and the size of the spare video memory area.

4. The method according to claim 1, characterized in that: Determining a first video memory region for key-value caching includes: Based on the model for inference, determining model parameters; Based on the model parameters, calculating the video memory block size; and The first video memory area is divided into blocks based on the video memory block size.

5. The method according to claim 1, characterized in that The first video memory area has a first video memory type, and the second video memory area has a second video memory type, wherein the first video memory type and the second video memory type are different.

6. A method for accessing a video memory managed by the method according to any one of claims 1 to 5, characterized in that: include: Determine whether the input parameters of the current input are the same as the input parameters of the previous input; In response to the input parameter of the current input being the same as the input parameter of the previous input, determining whether the first video memory area satisfies a threshold condition; In response to the first video memory region satisfying a threshold condition, the video memory is accessed based on a current configuration of a global video memory region.

7. The method according to claim 6, characterized in that Also includes: In response to the input parameter of the current input being different from the input parameter of the previous input, releasing the current second video memory area, converting the current second video memory area into the empty video memory area to obtain a new empty video memory area; Determining the size of the new second video memory area based on the currently inputted input parameters; Comparing the size of the new second video memory area with the size of the new empty video memory area; as well as In response to the size of the new second video memory area being smaller than or equal to the size of the new empty video memory area, the new second video memory area is allocated to the second end of the global video memory area.

8. The method according to claim 7, characterized in that Also includes: In response to the size of the new second video memory area being larger than the size of the new empty video memory area, determining whether the first video memory area includes an unused area; In response to the first video memory area including an unused area, releasing unused video memory blocks in the unused area, so as to convert the video memory area corresponding to the unused video memory blocks into the empty video memory area, so as to obtain an expanded new empty video memory area; Compare the size of the new second video memory area with the size of the expanded new empty video memory area; as well as In response to the size of the new second video memory area being smaller than or equal to the expanded new empty video memory area, the new second video memory area is allocated to the second end of the global video memory area.

9. A device for managing video memory, characterized in that: include: A first video memory manager is configured to manage a first video memory area for key-value caching; A second video memory manager is configured to manage a second video memory area for activating a value cache; as well as The global video memory manager is configured to: At the board, determining a global video memory area for the board; Allocating the first video memory area in the global video memory area; and Allocating the second video memory area in the global video memory area, Wherein, the current size of at least one of the first video memory area and the second video memory area is changed by releasing at least a portion of at least one of the first video memory area and the second video memory area to an unoccupied empty video memory area in the global video memory area, or converting at least a portion of the empty video memory area into at least one of the first video memory area and the second video memory area.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a machine, the method according to any one of claims 1 to 8 is performed.

11. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a machine, performs the method according to any one of claims 1 to 8.

Images