Resource management method and apparatus, and electronic device

Abstract

This disclosure provides a resource management method, apparatus, and electronic device, relating to the field of artificial intelligence technology, particularly deep learning and large model technologies. The specific implementation scheme is as follows: receiving a resource reallocation request; obtaining tensor information of a target tensor from a tensor dictionary; allocating a corresponding first storage space for the target tensor on the GPU based on the tensor information; and pointing a pointer object of the target tensor to the first physical storage address of the first storage space. Thus, without relying on a unified virtual address, it achieves the allocation of a corresponding first storage space for the target tensor on the GPU and points the pointer object of the target tensor to the first physical storage address of the first storage space. This allows the GPU kernel to quickly obtain the physical storage address of the target tensor on the GPU based on the pointer object, without needing to perform a virtual address to physical address conversion step, thereby improving the access efficiency of tensor data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning and large models, and especially to a resource management method, apparatus and electronic device. Background Technology

[0002] Currently, tensor data for large language models is typically stored using Graphics Processing Units (GPUs). However, as the model size continues to increase, the total memory usage of tensors quickly exceeds the memory capacity of a single GPU, leading to a severe resource bottleneck in storing all tensors on the GPU.

[0003] To alleviate memory pressure, related technologies typically employ temporarily storing some tensors in the Central Processing Unit (CPU) and reloading them back to the GPU when needed for subsequent computations. Currently, during the reloading process, the GPU's Memory Management Unit (MMU) detects that its local page table lacks a valid mapping between the virtual address and physical memory address of the corresponding tensor, triggering a page fault. In response to this fault, the Unified Memory driver migrates the tensor data from the CPU to the GPU's physical memory, updates the GPU sidepage table, establishes a mapping between the virtual address and the newly allocated physical memory address, and marks the physical memory address corresponding to the tensor's virtual address in the CPU's mapping table as invalid. This allows the GPU kernel to obtain the tensor's physical memory address and complete the access based on the updated GPU sidepage table. However, this approach relies on the Unified Memory (UM) mechanism, which is based on Unified Virtual Addressing (UVA) technology. UVA allocates a single virtual address valid in both the CPU and GPU address spaces, enabling transparent access and automatic migration of tensor data. However, if the CPU and GPU do not support such a single memory mechanism, the required tensors cannot be transparently reloaded from host memory to GPU memory at runtime, causing the GPU kernel to be unable to obtain the corresponding tensor data of the model due to accessing invalid addresses. Summary of the Invention

[0004] This disclosure provides a resource management method, apparatus, and electronic device.

[0005] According to one aspect of this disclosure, a resource management method is provided, the method comprising: receiving a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for a target tensor of a target model on a graphics processing unit (GPU); obtaining tensor information of the target tensor from a tensor dictionary; allocating a corresponding first storage space for the target tensor on the GPU according to the tensor information; and pointing a pointer object of the target tensor to a first physical storage address of the first storage space.

[0006] According to another aspect of this disclosure, a resource management apparatus is provided, the apparatus comprising: a receiving module, configured to receive a resource reallocation request, wherein the resource reallocation request is configured to request the reallocation of storage space for a target tensor of a target model on a graphics processing unit (GPU); an obtaining module, configured to obtain tensor information of the target tensor from a tensor dictionary; an allocation module, configured to allocate a corresponding first storage space for the target tensor on the GPU according to the tensor information; and a control module, configured to point a pointer object of the target tensor to a first physical storage address of the first storage space.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: a central processing unit (CPU), a graphics processing unit (GPU), a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the CPU and the GPU, the CPU, the GPU, and the storage medium communicate with each other via the bus, and the GPU executes the machine-readable instructions to perform the resource management method proposed above in this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the resource management method proposed in this disclosure above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the resource management method proposed above in this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure; Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure; Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure; Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure; Figure 7 This is a block diagram of an electronic device used to implement the resource management method of the embodiments of this disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] To alleviate memory pressure, related technologies typically involve temporarily storing some tensors in the Central Processing Unit (CPU) and reloading them back to the GPU when needed for subsequent computations. Currently, during the process of reloading the corresponding tensor back to the GPU, the graphics processor's Memory Management Unit (MMU) detects that its local page table does not yet have a valid mapping between the virtual address and physical memory address of the corresponding tensor, thus triggering a page fault. In response to this fault, the Unified Memory driver migrates the tensor data from the CPU to the GPU's physical memory, updates the GPU sidepage table, establishes a mapping from virtual address to the newly allocated physical memory address, and marks the physical memory address corresponding to the tensor's virtual address in the CPU's mapping table as an invalid address. Thus, the GPU kernel can obtain the tensor's physical memory address based on the updated GPU sidepage table and complete the access. However, this approach relies on the Unified Memory (UM) mechanism, which is based on Unified Virtual Addressing (UVA) technology. By allocating a single virtual address valid in both the CPU and GPU address spaces, it achieves transparent access and automatic migration of tensor data. However, if the CPU and GPU do not support such a single memory mechanism, the required tensors cannot be transparently reloaded from the host memory to the GPU memory at runtime. This causes the GPU kernel to be unable to obtain the corresponding tensor data of the model due to accessing invalid addresses. Therefore, how to provide a resource management method that does not rely on a unified virtual address is a technical problem that urgently needs to be solved.

[0015] To address the aforementioned problems, this disclosure proposes a resource management method, apparatus, and electronic device.

[0016] Figure 1 The diagram is based on the first embodiment of this disclosure. It should be noted that the resource management method of this disclosure can be applied to a resource management device, which can be a memory manager or can be configured in a memory manager.

[0017] This memory manager can be configured in electronic devices.

[0018] It should be noted that the electronic device in this embodiment may also include a graphics processor.

[0019] like Figure 1 As shown, this resource management method may include the following steps: Step 101: Receive a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU).

[0020] In this embodiment, the target model can be any type of large model. For example, the target model can be a multimodal large model, a large language model that supports long context reasoning, or a question-answering large language model, etc. This embodiment does not make any specific limitations on this.

[0021] Here, the target tensor is any tensor of the target model.

[0022] In some embodiments, for any tensor of the target model, if it is determined that the tensor will be accessed by the GPU and that the tensor does not have corresponding storage space in the GPU, then the tensor is taken as the target tensor, a resource reallocation request for the target tensor is generated, and the resource allocation request is sent to the resource management device, which receives the resource allocation request accordingly.

[0023] In some embodiments, one possible implementation of determining that the tensor does not have a corresponding storage space in the GPU is: if it is determined that the pointer object corresponding to the tensor points to a placeholder tensor with zero elements, then it is determined that the tensor does not have a corresponding storage space in the GPU.

[0024] Step 102: Obtain the tensor information of the target tensor from the tensor dictionary.

[0025] In this embodiment, tensor information refers to relevant information necessary for allocating storage space for the target tensor, such as the metadata of the target tensor or the physical storage address of the target tensor on the CPU.

[0026] The metadata of the target tensor may include: the data type, shape, and device type of the target tensor.

[0027] Step 103: Based on the tensor information, allocate the corresponding first storage space for the target tensor on the GPU.

[0028] In some embodiments, when the tensor information includes the metadata of the target tensor, the storage space required by the target tensor can be determined based on the metadata, and a corresponding first storage space can be allocated for the target tensor on the GPU based on the storage space required by the tensor.

[0029] Step 104: Point the pointer object of the target tensor to the first physical storage address of the first storage space.

[0030] In other words, in this embodiment, after allocating the corresponding first storage space for the target tensor on the GPU, the first physical storage address of the first resource is assigned to the pointer object of the target tensor, so that the GPU kernel can obtain the first physical storage address of the target tensor through the pointer object of the target tensor and access the first physical storage address. For example, the tensor value of the target tensor can be stored in the storage space corresponding to the first physical storage address according to the first physical storage address, or the tensor value of the target tensor can be obtained from the first physical storage address, etc. This embodiment does not make specific limitations in this regard.

[0031] The resource management method of this disclosure receives a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for a target tensor of a target model on a graphics processing unit (GPU); obtains tensor information of the target tensor from a tensor dictionary; allocates a corresponding first storage space for the target tensor on the GPU based on the tensor information; and points a pointer object of the target tensor to a first physical storage address of the first storage space. Thus, without relying on a unified virtual address, it achieves the allocation of a corresponding first storage space for the target tensor on the GPU and points the pointer object of the target tensor to the first physical storage address of the first storage space. This allows the GPU kernel to quickly obtain the physical storage address of the target tensor on the GPU based on the pointer object, without needing to perform a virtual address to physical address conversion step, thereby helping to improve the access efficiency of tensor data.

[0032] Figure 2 This is a schematic diagram based on the second embodiment of the present disclosure. Figure 2 The illustrated embodiment may include the following steps: Step 201: Receive a first resource release request, wherein the first resource release request is used to request the release of the storage space allocated for the target tensor on the GPU.

[0033] Step 202: Obtain the second physical storage address pointed to by the pointer object of the target tensor.

[0034] Step 203: Release the second storage space corresponding to the second physical storage address, wherein the second storage space is located on the GPU.

[0035] Step 204: Set the pointer object to point to a placeholder tensor with zero elements.

[0036] Step 205: Receive a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU).

[0037] Step 206: Obtain the tensor information of the target tensor from the tensor dictionary.

[0038] Step 207: Based on the tensor information, allocate the corresponding first storage space for the target tensor on the GPU.

[0039] Step 208: Point the pointer object of the target tensor to the first physical storage address of the first storage space.

[0040] It should be noted that for a detailed description of steps 205 to 208, please refer to the relevant descriptions in other embodiments, which will not be repeated here.

[0041] In this embodiment, before receiving a resource reallocation request, a first resource release request for the target tensor can also be received. Based on the first resource release request, the second physical storage address pointed to by the pointer object is obtained, and the second storage space corresponding to the second physical storage address is released. The pointer object is then set to point to a placeholder tensor with zero elements. Thus, combined with the first resource release request, the storage space occupied by the target tensor on the GPU is released promptly, thereby improving the storage space reuse efficiency on the GPU and avoiding subsequent allocation failures or waiting due to long-term tensor occupation.

[0042] To facilitate a clear understanding of this disclosure, the resource management method of this embodiment will be described exemplarily below, with the target tensor's tensor type being the weight tensor type and the tensor information including the target tensor's tensor value at the third physical storage address in the central processing unit (CPU).

[0043] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure.

[0044] like Figure 3 As shown, the method may further include: Step 301: Receive a first resource release request, wherein the first resource release request is used to request the release of the storage space allocated for the target tensor on the GPU.

[0045] Step 302: Obtain the second physical memory address pointed to by the pointer object.

[0046] Step 303: Store the tensor value of the target tensor in the second physical storage address into the third physical storage address of the CPU.

[0047] It should be noted that the CPU described above can be located in an electronic device; that is, the electronic device in this embodiment may also include a CPU.

[0048] Step 304: Save the third physical storage address of the target tensor in the tensor dictionary.

[0049] In this embodiment, when the target tensor is of type weight tensor, before releasing the second storage space corresponding to the second physical storage address, the tensor value of the target tensor in the second physical storage address is stored in the third physical storage address of the CPU, and the third physical storage address of the target tensor is saved in the tensor information. This makes it convenient to obtain the tensor value of the target tensor based on the third physical storage address in the tensor dictionary when it is necessary to reallocate the corresponding storage space for the target tensor on the GPU later, and reallocate the corresponding storage space for the target tensor on the GPU based on the tensor value.

[0050] Step 305: Release the second storage space corresponding to the second physical storage address, wherein the second storage space is on the GPU.

[0051] Step 306: Set the pointer object to point to a placeholder tensor with zero elements.

[0052] Step 307: Receive a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU).

[0053] Step 308: Obtain the third physical storage address of the target tensor from the tensor dictionary.

[0054] Step 309: Obtain the tensor value of the target tensor from the third physical memory address of the CPU.

[0055] Step 310: Based on the tensor value, allocate a corresponding first storage space for the target tensor on the GPU, and store the tensor value in the first storage space.

[0056] In some embodiments, the storage space required by the target tensor is determined based on the tensor value, and a corresponding first storage space is allocated for the target tensor on the GPU based on the storage space required by the target tensor. This ensures that the allocated first storage space matches the storage space required by the target tensor.

[0057] In this embodiment, when the target tensor is of type weight tensor, the target tensor's tensor value stored on the CPU is combined with the tensor value stored on the CPU to accurately allocate the corresponding first storage space on the GPU, so that the allocated first storage space matches the storage space requirement of the target tensor, thereby avoiding space waste and maximizing resource utilization.

[0058] Step 311: Point the pointer object of the target tensor to the first physical storage address of the first storage space.

[0059] Step 312: Delete the tensor value in the third physical memory address of the CPU and release the memory space corresponding to the third physical memory address.

[0060] In this embodiment, after storing the target tensor value from the CPU to the first storage space of the GPU and setting the pointer object of the target tensor to the first physical storage address in the first storage space, the tensor value in the third physical storage address of the CPU is promptly deleted, and the storage space corresponding to the third physical storage address is released. Therefore, by promptly releasing the storage space occupied by the target tensor value on the CPU after storing it from the CPU to the first storage space of the GPU, the reuse efficiency of the CPU's storage space is improved.

[0061] To facilitate a clear understanding of this disclosure, the resource management method of this embodiment will be described exemplarily below, with the target tensor's tensor type being the key-value cache tensor type and the tensor information including the target tensor's first meta-information.

[0062] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure.

[0063] like Figure 4 As shown, the method may further include: Step 401: Receive a first resource release request, wherein the first resource release request is used to request the release of the storage space allocated for the target tensor on the GPU.

[0064] Step 402: Obtain the second physical memory address pointed to by the pointer object.

[0065] Step 403: Release the second storage space corresponding to the second physical storage address, wherein the second storage space is located on the GPU.

[0066] Step 404: Save the first element information of the target tensor in the tensor dictionary.

[0067] The first element information may include: the first data type, the first shape, and the first device type of the target tensor.

[0068] In some embodiments, the first metadata of the target tensor may be obtained by the resource management device from the first resource release request (i.e., the first resource release request may include the first metadata of the target tensor), or it may be obtained by the resource management device through communication with other units. This embodiment does not specifically limit the method of obtaining the first metadata of the target tensor.

[0069] In this embodiment, the first metadata of the target tensor is stored in the tensor dictionary, so that when it is necessary to allocate corresponding storage space for the target tensor on the GPU, the corresponding storage space can be allocated on the GPU based on the first metadata of the target tensor in the tensor dictionary.

[0070] Step 405: Set the pointer object to point to a placeholder tensor with zero elements.

[0071] Step 406: Receive a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU).

[0072] Step 407: Obtain the first element information of the target tensor from the tensor dictionary.

[0073] Step 408: Based on the first element information, allocate the corresponding first storage space for the target tensor on the GPU.

[0074] In some embodiments, a corresponding first storage space can be allocated on the GPU for the target tensor, wherein the first storage space corresponds to the first metadata. Thus, the corresponding first storage space can be accurately allocated on the GPU, ensuring that the allocated first storage space matches the storage space requirements of the target tensor, thereby avoiding space waste and maximizing resource utilization.

[0075] Step 409: Point the pointer object of the target tensor to the first physical storage address of the first storage space.

[0076] In this embodiment, during the reasoning process of the target model, the tensor value of the target tensor can be written to the first physical storage address.

[0077] Step 410: Delete the first element information from the tensor dictionary.

[0078] It is understood that when the target tensor is of the KV cache tensor type, it means that the target tensor is a KV cache tensor type tensor. For tensors of this type, new tensor values ​​of this type will be generated during the inference process of the target model. Therefore, the metadata of tensors of this type is changing. Therefore, in this embodiment, after pointing the pointer object of the target tensor to the first physical storage address of the first storage space, the first metadata of the target tensor in the tensor dictionary is deleted, thereby avoiding the impact of the historical metadata of the target tensor on the subsequent resource management of the target tensor, thus preparing for the next resource release of the target tensor.

[0079] Based on any of the above embodiments, when the GPU temporarily does not need to access the first storage space of the target tensor, the method may further include: receiving a second resource release request, the second resource release request being used to request the release of the first storage space; releasing the first storage space; and setting the pointer object to point to a placeholder tensor with zero elements. Thus, timely release of the first storage space, allowing it to be allocated to other tensors, helps improve the reuse rate of GPU storage resources.

[0080] In some embodiments, when the target tensor is of type weight tensor and the tensor information includes the tensor value of the target tensor at a third physical storage address in the central processing unit (CPU), before releasing the first storage space, the tensor value of the target tensor in the first storage space may be stored at a fourth physical storage address on the CPU, and the third physical storage address of the target tensor in the tensor dictionary may be replaced with the fourth physical storage address.

[0081] In other embodiments, when the target tensor is of KV tensor type, the first metadata in the tensor dictionary may be deleted before receiving the second resource release request. Correspondingly, after receiving the second resource release request, the method may further include: storing the second metadata of the target tensor in the tensor dictionary.

[0082] As an example, a second resource release request may include: second meta-information of the target tensor.

[0083] As another example, the resource management device can obtain second-dimensional information about the target tensor by interacting with other units.

[0084] The second element information may include: the second data type, second shape, and second device type of the target tensor, etc., but this embodiment does not specifically limit this.

[0085] To facilitate a clear understanding of this disclosure, the following will be combined with... Figure 5 The resource management method of this embodiment is described by way of example.

[0086] Figure 5 This is a schematic diagram according to the fifth embodiment of this disclosure. It should be noted that this embodiment uses the example of the resource management method being executed by a memory manager for illustrative purposes.

[0087] like Figure 5 As shown, the method may further include: Step 501: Obtain the target tensor of the target model and the tensor type of the target tensor.

[0088] Step 502: If the tensor type is a KV cache tensor type, save the initial metadata of the target tensor in the tensor dictionary, and allocate the corresponding initial storage space for the target tensor on the GPU according to the initial metadata.

[0089] In some embodiments, the pointer object of the target tensor can also be pointed to the storage address of the initial storage space, so that the CPU core can obtain the storage address of the target tensor on the GPU through the pointer object, omitting the address mapping and helping to improve the access efficiency of the target tensor.

[0090] Step 503: Determine whether the initial storage space of the target tensor needs to be released. If so, proceed to step 504. It should be noted that if not, the next step is to determine whether the initial storage space of the target tensor needs to be released.

[0091] Step 504: Replace the initial metadata of the target tensor in the tensor dictionary with the first metadata of the target tensor.

[0092] Step 505: Set the pointer object of the target tensor to point to a placeholder tensor with zero elements.

[0093] As an example, the tensor dictionary can be traversed through the release function to obtain the initial metadata of the target tensor from the tensor dictionary, and the initial metadata of the target tensor in the tensor dictionary can be replaced with the first metadata of the target tensor. The pointer object of the target tensor is then set to point to a placeholder tensor with zero elements.

[0094] Step 506: Release the initial storage space so that it can be allocated to other tensors.

[0095] As an example, Python's garbage collection mechanism and PyTorch's memory cache cleanup interface can be used to free up initial storage space.

[0096] Step 507: Determine whether it is necessary to reallocate the corresponding storage space for the target tensor on the GPU. If so, proceed to step 508.

[0097] It should be noted that if not, the next step is to determine whether the corresponding storage space needs to be reallocated for the target tensor on the GPU.

[0098] Step 508: Obtain the first element information of the target tensor from the tensor dictionary.

[0099] Step 509: Based on the first element information, reallocate the corresponding first storage space for the target tensor on the GPU.

[0100] Step 510: Assign the storage address of the first storage space to the pointer object of the target tensor.

[0101] Step 511: Delete the first element information of the target tensor from the tensor dictionary in order to prepare for the next resource release of the target tensor.

[0102] Step 512: If the target tensor is of type weight tensor, store the tensor information of the target tensor in the tensor dictionary, allocate the corresponding initial storage space for the target tensor on the GPU, and store the tensor value of the target tensor in the initial storage space.

[0103] The tensor information may include metadata about the target tensor.

[0104] In some embodiments, the pointer object of the target tensor can also be pointed to the storage address corresponding to the initial storage space.

[0105] Step 513: Determine whether the initial storage space of the target tensor needs to be released. If so, proceed to step 514.

[0106] It should be noted that if not, the next step is to determine whether the initial storage space of the target tensor needs to be released.

[0107] Step 514: Copy the tensor value of the target tensor in the initial storage space to the CPU's third storage space.

[0108] Step 515: Replace the tensor information of the target tensor in the tensor dictionary with the third physical storage address, and set the pointer object of the target tensor to point to a placeholder tensor with zero elements, where the third physical storage address is the physical storage address of the third storage space.

[0109] Step 516: Release the initial storage space so that it can be allocated to other tensors.

[0110] Step 517: Determine whether it is necessary to reallocate the corresponding storage space for the target tensor on the GPU. If so, proceed to step 518.

[0111] It should be noted that if not, the next step is to determine whether the corresponding storage space needs to be reallocated for the target tensor on the GPU.

[0112] Step 518: Obtain the third physical storage address of the target tensor from the tensor dictionary.

[0113] Step 519: Obtain the tensor value of the target tensor from the third physical storage address, allocate a corresponding first storage space for the target tensor on the GPU based on the tensor value, and store the tensor value of the target tensor in the first storage space.

[0114] Step 520: Assign the storage address of the first storage space to the pointer object of the target tensor.

[0115] Step 521: Delete the tensor value in the CPU's third physical memory address and release the memory space corresponding to the third physical memory address.

[0116] This embodiment provides a resource management method that does not rely on a unified memory address. It realizes the direct binding between the pointer object of the target tensor and the physical storage address, without the need to perform the virtual address to physical address conversion step, thereby reducing management complexity and address maintenance overhead.

[0117] To implement the above embodiments, this disclosure also provides a resource management device. For example... Figure 6 As shown, Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure.

[0118] The resource management device 60 may include: a receiving module 601, an acquisition module 602, an allocation module 603, and a control module 604.

[0119] The receiving module 601 is used to receive a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU).

[0120] The acquisition module 602 is used to obtain the tensor information of the target tensor from the tensor dictionary.

[0121] The allocation module 603 is used to allocate the corresponding first storage space for the target tensor on the GPU according to the tensor information.

[0122] The control module 604 is used to point the pointer object of the target tensor to the first physical storage address of the first storage space.

[0123] As one possible implementation of this disclosure, the apparatus may further include: The first resource release module is used to receive a first resource release request, wherein the first resource release request is used to request the release of the storage space allocated for the target tensor on the GPU; obtain the second physical storage address pointed to by the pointer object; release the second storage space corresponding to the second physical storage address, wherein the second storage space is on the GPU; and set the pointer object to point to a placeholder tensor with zero elements.

[0124] As one possible implementation of this disclosure, when the target tensor is of type weight tensor, the tensor information includes: the third physical storage address of the target tensor value in the central processing unit (CPU). The allocation module 603 is specifically used to: obtain the tensor value of the target tensor from the third physical storage address of the CPU; and allocate a corresponding first storage space for the target tensor on the GPU according to the tensor value, wherein the first storage space is used to store the tensor value.

[0125] As one possible implementation of this disclosure, the apparatus may further include: The first deletion module is used to delete the tensor value from the third physical memory address of the CPU; The first release module is used to release the storage space corresponding to the third physical storage address.

[0126] As one possible implementation of this disclosure, the apparatus may further include: The first storage module is used to store the tensor value of the target tensor in the second physical storage address into the third physical storage address of the CPU; and to store tensor information in the tensor dictionary.

[0127] As one possible implementation of this disclosure, when the target tensor is a key-value cache tensor, the tensor information includes: the first meta-information of the target tensor; The allocation module 603 is specifically used to: allocate a corresponding first storage space for the target tensor on the GPU based on the first element information.

[0128] As one possible implementation of this disclosure, the apparatus may further include: The second storage module is used to store the first element information in the tensor dictionary.

[0129] As one possible implementation of this disclosure, the apparatus may further include: The second deletion module is used to delete the first element information from the tensor dictionary.

[0130] As one possible implementation of this disclosure, the apparatus may further include: The second resource release module is used to receive a second resource release request, which requests the release of the first storage space; release the first storage space; and set the pointer object to point to a placeholder tensor with zero elements.

[0131] It should be noted that the foregoing explanation of the resource method embodiment also applies to the resource management device of this embodiment, and this embodiment does not specifically limit it in this regard.

[0132] The resource management device of this disclosure receives a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for a target tensor of a target model on a graphics processing unit (GPU); obtains tensor information of the target tensor from a tensor dictionary; allocates a corresponding first storage space for the target tensor on the GPU according to the tensor information; and points a pointer object of the target tensor to a first physical storage address of the first storage space. Thus, without relying on a unified virtual address, it achieves the allocation of a corresponding first storage space for the target tensor on the GPU and points the pointer object of the target tensor to the first physical storage address of the first storage space. This allows the GPU kernel to quickly obtain the physical storage address of the target tensor on the GPU based on the pointer object, without performing a virtual address to physical address conversion step, thereby helping to improve the access efficiency of tensor data.

[0133] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, are all carried out with the user's consent and comply with relevant laws and regulations, and do not violate public order and good morals.

[0134] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0135] Figure 7 This is a block diagram of an electronic device used to implement the resource management method of the embodiments of this disclosure.

[0136] like Figure 7 As shown, the electronic device 700 includes: a central processing unit 701, a graphics processor 702, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the central processing unit 701 and the graphics processor 702. The central processing unit 701, the graphics processor 702, and the storage medium communicate with each other via the bus. The central processing unit 701 and the graphics processor 702 execute the machine-readable instructions to perform the steps of the resource management method as described in the following embodiments.

[0137] The central processing unit 701, graphics processor 702, and various bus components are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The resource management device includes at least one software function module that can be stored in a storage medium or embedded in the operating system (OS) of the electronic device 70 in the form of software or firmware. The central processing unit 701 and graphics processor 702 are used to execute executable modules stored in the storage medium, such as the software function modules and computer programs included in the resource management device.

[0138] It should be noted that the electronic device 700 in this embodiment may also include a memory manager, although no example of a memory manager is given in this embodiment.

[0139] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A resource management method, comprising: Receive a resource reallocation request, wherein the resource reallocation request is used to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU). Obtain the tensor information of the target tensor from the tensor dictionary; Based on the tensor information, allocate a corresponding first storage space for the target tensor on the GPU; Point the pointer object of the target tensor to the first physical storage address of the first storage space.

2. The method according to claim 1, wherein, Before receiving the resource reallocation request, the method further includes: Receive a first resource release request, wherein the first resource release request is used to request the release of the storage space allocated on the GPU for the target tensor; Obtain the second physical memory address pointed to by the pointer object; Release the second storage space corresponding to the second physical storage address, wherein the second storage space is located on the GPU; Set the pointer object to point to a placeholder tensor with zero elements.

3. The method according to claim 2, wherein, When the target tensor is of type weight tensor, the tensor information includes: the third physical storage address of the tensor value of the target tensor in the central processing unit (CPU). Wherein, allocating a corresponding first storage space for the target tensor on the GPU based on the tensor information includes: Obtain the tensor value of the target tensor from the third physical memory address of the CPU; Based on the tensor value, a corresponding first storage space is allocated on the GPU for the target tensor, wherein the first storage space is used to store the tensor value.

4. The method according to claim 3, wherein, The method further includes: Delete the tensor value from the third physical memory address of the CPU; Release the storage space corresponding to the third physical storage address.

5. The method according to claim 3, wherein, Before releasing the second storage space corresponding to the second physical storage address, the method further includes: Store the tensor value of the target tensor in the second physical storage address into the third physical storage address of the CPU; The tensor information is stored in the tensor dictionary.

6. The method according to claim 2, wherein, When the target tensor is of key-value (KV) cache tensor type, the tensor information includes: the first meta-information of the target tensor; Wherein, allocating a corresponding first storage space for the target tensor on the GPU based on the tensor information includes: Based on the first metadata, a corresponding first storage space is allocated on the GPU for the target tensor.

7. The method according to claim 6, wherein, The method further includes: The first metadata is stored in the tensor dictionary.

8. The method according to claim 6, wherein, After pointing the pointer objects in the tensor dictionary to the first physical storage address of the first storage space, the method includes: Delete the first meta-information from the tensor dictionary.

9. The method according to claim 1, wherein, The method further includes: Receive a second resource release request, the second resource release request being used to request the release of the first storage space; Release the first storage space; Set the pointer object to point to a placeholder tensor with zero elements.

10. A resource management device, comprising: A receiving module is configured to receive a resource reallocation request, wherein the resource reallocation request is configured to request the reallocation of storage space for the target tensor of the target model on the graphics processing unit (GPU). The acquisition module is used to obtain the tensor information of the target tensor from the tensor dictionary; An allocation module is used to allocate a corresponding first storage space for the target tensor on the GPU according to the tensor information; The control module is used to point the pointer object of the target tensor to the first physical storage address of the first storage space.

11. An electronic device, comprising: A central processing unit (CPU), a graphics processing unit (GPU), a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the CPU and the GPU, the CPU, the GPU, and the storage medium communicate with each other via the bus, and the GPU executes the machine-readable instructions to perform the method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images