Storage management method, storage mapping method, device and system of acceleration equipment

Abstract

The invention discloses a storage management method, a storage mapping method, a storage management device and a storage mapping system for acceleration equipment. The storage management method of the acceleration equipment is executed by a CPU (Central Processing Unit), and comprises the following steps of: in global storage of the acceleration equipment, allocating a memory pool for an AI program, and executing the AI program; in the process of executing the memory allocation instruction, allocating memory blocks for the data parameters in the memory pool according to the annular linked list; when the device kernel function is started, a device kernel function starting request is sent to an acceleration device in the AI computing system, and the base address of the memory pool and the global address pointer of each data parameter in the device kernel function are transmitted into the stub function, so that the acceleration device executes the stub function to start the device kernel function. And acquiring a virtual address of each data parameter in the address space of the acceleration equipment, and performing memory access operation according to the virtual address of each data parameter in the process of executing the kernel function of the equipment. By adopting the technical scheme, the memory allocation in the acceleration equipment can be optimized, and the overall performance of the system is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence chip technology, and in particular to storage management methods, storage mapping methods, devices and systems for acceleration devices. Background Technology

[0002] In artificial intelligence or scientific computing scenarios, device storage is often exclusively used by the runtime system, with data shared in parallel among parallel processors. With the development of heterogeneous computing technology, CPUs and acceleration devices can collaborate to complete AI tasks. However, when acceleration devices perform computational tasks, frequent device storage allocation and release operations lead to storage management overhead, storage fragmentation, and low storage utilization, thus affecting the overall system efficiency.

[0003] Furthermore, the device storage of heterogeneous acceleration devices similar to GPUs (Graphics Processing Units) is often multi-level. When multiple processor units in the acceleration device implement global device storage, they often need to map the global device storage address space to the local address space of the processor element (PE) to perform operations.

[0004] In existing address space mapping technologies, address mapping is generally performed through a relatively complex MMU (Hardware Memory Management Unit). The MMU usually contains a Fast Lookup Table (TLB), which records the mapping information of some addresses. However, the TLB usually has limited capacity, so the address mappings it can hold are also limited. Moreover, the mapped address is only returned immediately when the TLB is hit. Otherwise, the MMU needs to obtain the mapped address by querying the page table in the device memory, which often results in an overhead of hundreds or even thousands of clock cycles. Existing technologies have high overhead for applications that exclusively occupy device resources. Summary of the Invention

[0005] This invention provides a storage management method, storage mapping method, apparatus, and system for acceleration devices, which can optimize memory allocation in acceleration devices for AI (Artificial Intelligence) application scenarios and improve the overall system performance.

[0006] According to one aspect of the present invention, a storage management method for an acceleration device is provided, executed by a CPU in an AI computing system, comprising:

[0007] In the global storage of the acceleration device, a memory pool is allocated for the AI ​​program, and the AI ​​program is executed; wherein, the memory pool is managed in the form of a circular linked list;

[0008] During the execution of memory allocation instructions, memory blocks are allocated for data parameters in the memory pool according to the circular linked list; wherein, the memory blocks are represented by global address pointers;

[0009] When starting the device kernel function, a device kernel function start request is sent to the acceleration device in the AI ​​computing system. The base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform memory access operations based on the virtual address of each data parameter during the execution of the device kernel function.

[0010] According to another aspect of the present invention, a storage mapping method for an acceleration device is provided, executed by an acceleration device in an AI computing system, comprising:

[0011] In response to the CPU's request to start the device kernel function, the execution of the stub function is triggered; wherein, during the compilation of the executable file, the stub function is used in advance to replace the device kernel function as the program entry point, and the call to the device kernel function is generated in the stub function;

[0012] Based on the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the storage management unit, the pointer parameters of each data parameter are modified to obtain the virtual address of each data parameter in the address space of the acceleration device; wherein, the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the acceleration device described in any embodiment of the present invention and passed into the stub function; the address offset in the storage management unit is pre-configured during the compilation process of the executable file;

[0013] During the execution of the device kernel function, the memory management unit is invoked to perform virtual and physical address mapping on each data accessed in the memory access instruction based on the address offset and the virtual address of each data parameter, so as to obtain the physical address of each data accessed in the memory access instruction in the global storage, and to perform the memory access operation based on the physical address of each data accessed.

[0014] According to another aspect of the present invention, a storage management apparatus for an acceleration device is provided, executed by a CPU in an AI computing system, comprising:

[0015] The memory pool allocation module is used to allocate a memory pool for the AI ​​program in the global storage of the acceleration device and execute the AI ​​program; wherein the memory pool is managed in the form of a circular linked list;

[0016] The memory block allocation module is used to allocate memory blocks for data parameters in the memory pool according to the circular linked list during the execution of memory allocation instructions; wherein, the memory blocks are represented by global address pointers;

[0017] The device kernel function startup module is used to send a device kernel function startup request to the acceleration device in the AI ​​computing system during the execution of the startup instruction for the device kernel function. It also passes the base address of the memory pool and the global address pointers of each data parameter in the device kernel function to the stub function, so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform data parameter memory access operations based on the virtual address of each data parameter during the execution of the device kernel function.

[0018] According to another aspect of the present invention, a storage mapping apparatus for an acceleration device is provided, executed by an acceleration device in an AI computing system, comprising:

[0019] The stub function startup module is used to respond to the CPU's startup request for the device kernel function and trigger the execution of the stub function; wherein, during the compilation process of the executable file, the stub function is used in advance to replace the device kernel function as the program entry point, and the stub function generates the call to the device kernel function;

[0020] The pointer parameter correction module is used to correct the pointer parameters of each data parameter according to the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the storage management unit, so as to obtain the virtual address of each data parameter in the address space of the acceleration device; wherein, the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the acceleration device according to any embodiment of the present invention and passed into the stub function; the address offset in the storage management unit is pre-configured during the compilation process of the executable file;

[0021] The address mapping module is used to perform virtual and physical address mapping on the data involved in the memory access instruction through the storage management unit and based on the address offset and the virtual address of each data parameter during the execution of the device kernel function. This is to obtain the physical address of each accessed data in the global storage and to perform the memory access operation based on the physical address of each accessed data.

[0022] According to another aspect of the present invention, an AI computing system is provided, the AI ​​computing system comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to execute the storage management method of the acceleration device described in Embodiment 1 and Embodiment 2 of the present invention, and the storage mapping method of the acceleration device described in Embodiment 3 of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the storage management method of the acceleration device described in Embodiment 1 and Embodiment 2 of the present invention, and the storage mapping method of the acceleration device described in Embodiment 3 of the present invention.

[0027] The technical solution of this invention allocates a memory pool for the AI ​​program in the global storage of the acceleration device and executes the AI ​​program. This allows for pre-allocation of the memory pool before task startup, enabling the acceleration device to reuse the memory during operation. This avoids requesting memory from the CPU for each calculation, ensuring the AI ​​program's long-term exclusive access to storage. By allocating memory blocks for data parameters in the memory pool according to a circular linked list during memory allocation instruction execution, the continuity of memory allocation in the memory pool can be guaranteed as much as possible, avoiding excessive memory fragmentation and improving the performance of shared memory allocation. When starting the device kernel function, a device kernel function startup request is sent to the acceleration device in the AI ​​computing system, and the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function. This allows the acceleration device to obtain the virtual address of each data parameter in the acceleration device's address space by executing the stub function, and to perform memory access operations based on the virtual address of each data parameter during the execution of the device kernel function. This reduces the clock cycles consumed by the MMU in the acceleration device for address mapping. By replacing TLB and page table lookups with simple address mapping calculations, the efficiency of data parameter read and write operations is improved, thereby improving AI computing performance.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1This is a flowchart of a storage management method for an acceleration device according to Embodiment 1 of the present invention;

[0031] Figure 2 This is a schematic diagram illustrating accessing global memory from a processor unit via an MMU in a prior art according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of a memory pool and memory block allocation according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of memory block allocation according to an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of another memory block allocation provided according to an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of another memory block allocation provided according to an embodiment of the present invention;

[0036] Figure 7 This is a flowchart of another storage management method for an acceleration device provided according to Embodiment 2 of the present invention;

[0037] Figure 8 This is a flowchart of a storage mapping method for an acceleration device according to Embodiment 3 of the present invention;

[0038] Figure 9 This is a schematic diagram of the structure of a storage management device for an acceleration device according to Embodiment 4 of the present invention;

[0039] Figure 10 This is a schematic diagram of the structure of a storage mapping device for an acceleration device according to Embodiment 5 of the present invention;

[0040] Figure 11 This is a schematic diagram of the structure of an AI computing system that implements the storage management method and storage mapping method of the acceleration device according to embodiments of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] Example 1

[0044] Figure 1 This is a flowchart of a storage management method for an acceleration device according to Embodiment 1 of the present invention. This embodiment is applicable to situations where memory allocation for an acceleration device is performed by a CPU. This method can be executed by a storage management device of the acceleration device, which can be implemented in hardware and / or software and is generally configured within a CPU with data processing capabilities. Figure 1 As shown, the method includes:

[0045] S110. Allocate a memory pool for the AI ​​program in the global storage of the acceleration device and execute the AI ​​program.

[0046] The memory pool is managed in the form of a circular linked list.

[0047] Understandably, with the development of heterogeneous computing technology, CPUs and acceleration devices can work together to complete AI task calculations. When acceleration devices perform calculation tasks, frequent device storage allocation and release operations lead to storage management overhead, storage fragmentation, and low storage utilization, which in turn affects the overall efficiency of the system. Therefore, a fixed-size contiguous memory pool can be pre-allocated before the task starts for the acceleration device to reuse during operation, avoiding requesting memory from the CPU for each calculation.

[0048] Optionally, in AI and scientific computing scenarios, device storage is often exclusively used by the runtime system and shared in parallel among parallel processors. In this scenario, a centralized allocation and release storage management method is more suitable. The AI ​​program is a heterogeneous AI device program, which is a program written in a single program with multiple data modes. The memory pool is allocated on a per-AI program basis, which can achieve high-performance shared global storage mapping during the execution of each AI program.

[0049] Figure 2This is a schematic diagram illustrating, in an alternative prior art, accessing global memory from a processor unit via an MMU. For example... Figure 2 As shown, the device processor unit is any processor unit that operates in parallel within the acceleration device. When executing a data memory access instruction, the processor unit needs to obtain the data request address based on the data address in the code stream. Then, it queries the TLB through the MMU to determine whether the data request address is hit in the TLB. If it is hit, the mapped data address is obtained from the TLB. Otherwise, the mapped data address needs to be obtained by querying the page table through the MMU. After obtaining the mapped data address, data read and write operations are performed in the device's global storage.

[0050] It can be seen that, Figure 2 In the existing technical solutions shown, the MMU needs to perform TLB lookups, and even when the TLB cannot be found, the page table needs to be consulted, resulting in an overhead of hundreds or even thousands of clock cycles. Figure 2 The device's global storage, as shown, contains a large number of scattered, allocated data blocks that occupy a significant amount of storage space. This large amount of memory fragmentation may reduce the efficiency of the device's global storage and memory allocation, making it impossible to store as much data as possible.

[0051] Optionally, acceleration devices can refer to hardware specifically designed to accelerate specific computing tasks, such as GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), GCUs, etc. Acceleration devices have strong parallel computing capabilities and are suitable for handling computationally intensive tasks such as AI training and inference. Global storage can refer to the shared memory space in acceleration devices, which can be accessed by all computing cores.

[0052] Optionally, the memory pool allocated in the global storage of the acceleration device can refer to a contiguous memory region in the global storage, used for dynamic allocation and reclamation of memory. Through memory pool management, the overhead caused by frequent memory allocation or memory release can be reduced, and the risk of memory fragmentation can be lowered.

[0053] Optionally, allocating a memory pool for the AI ​​program in the global storage of the acceleration device may include: determining the base address and size of the memory pool in the global storage of the acceleration device; for example, the base address of the memory pool may be determined to be 0x000100000000 in the global storage, and the size of the memory pool may be 1M, then the physical address of the actually allocated memory pool may be 0x000100000000~0x0001000FFFFF.

[0054] Optionally, a circular linked list is a data structure in which the last node of the linked list points to the head node, forming a closed loop. Taking the memory pool mentioned above as an example, when storage is allocated to 0x0001000FFFFF, the search is not stopped and storage management and other operations are not performed because 0x000100000000 is not used as the last address of the memory pool. Instead, the allocation continues from 0x000100000000.

[0055] S120. During the execution of memory allocation instructions, memory blocks are allocated for data parameters in the memory pool according to the circular linked list.

[0056] Memory blocks are represented by global address pointers.

[0057] It is understandable that the CPU allocates memory blocks for AI programs in the global device storage of the acceleration device; therefore, each memory block is represented by a global address pointer in the global device storage.

[0058] Optionally, data parameters can refer to data objects that need to be allocated space in the memory pool, such as inputs to AI calculations, intermediate calculation results, etc. Each data parameter has a specific memory size requirement.

[0059] The allocation of memory blocks for data parameters in the memory pool based on the circular linked list may include:

[0060] According to the memory allocation instruction, obtain the memory size required for the data parameters, and according to the circular linked list, obtain the starting allocation address of the memory pool;

[0061] Determine whether the size of the contiguous storage space in the memory pool, starting from the initial allocation address, is greater than or equal to the memory size required by the data parameters;

[0062] If so, then according to the starting allocation address and the memory size required by the data parameters, allocate a memory block for the data parameters in the memory pool and update the starting allocation address;

[0063] If not, then select the target free memory block according to the circular linked list, and allocate a memory block for the data parameters in the memory pool according to the target free memory block and the memory size required by the data parameters.

[0064] Optionally, the memory allocation instruction is a device storage management software interface called by the AI ​​program. For example, "p = deviceMalloc(p_size); q = deviceMalloc(q_size);" means calling the device storage allocation API to allocate memory for the data parameters p and q.

[0065] Optionally, when executing memory allocation instructions, the memory size required for each data parameter can be obtained, and then a memory block can be allocated for the data parameter in the free memory space of the circular linked list.

[0066] Figure 3 This is a diagram illustrating an optional memory pool and memory block allocation. (For example...) Figure 3 As shown, the circular region is a memory pool managed in the form of a circular linked list. The shaded area with diagonal lines represents allocated memory blocks, and the shaded area with dots represents freed memory blocks. Both freed and unallocated memory blocks are in a free state. Figure 3 The diagram also shows the current allocation point in the memory pool, which can be represented by the starting allocation address. Each memory block allocation continues from the current allocation point, that is, from the starting allocation address.

[0067] Optionally, the starting allocation address of the memory pool is maintained by a circular linked list.

[0068] Optionally, when allocating memory blocks for the first time in an empty memory pool, the base address of the memory pool can be used as the starting allocation address.

[0069] Figure 4 A diagram illustrating an optional memory block allocation. (Example) Figure 4 As shown, mb is the starting allocation address of the current record in the circular linked list, and ms is the memory size required by the data parameters, which is the size of the contiguous storage space required to allocate the memory block. After determining the starting allocation address, it is checked whether the size of the contiguous storage space after the starting allocation address can meet the memory size required by the parameters. If it can, the starting allocation address is updated to mb+ms to serve as the starting allocation address for the next memory block allocation, and the address space between mb and ms is used as the memory block for the data parameters.

[0070] Optionally, when the size of the contiguous storage space in the memory pool starting from the initial allocation address is less than the memory size required by the data parameters, a target free memory block is selected according to the circular linked list, including: searching for the first free memory block that meets the memory size requirement of the data parameters from the base address of the memory pool according to the circular linked list, and using it as the target free memory block.

[0071] Figure 5 A diagram illustrating the allocation of an alternative memory block. (See diagram below.) Figure 5As shown, when the contiguous storage space between the starting allocation address and the end address of the memory pool is insufficient to meet the memory requirements of the data parameters, if a data block of size ms is allocated for the data parameters after the starting allocation address, ms will exceed the end of the video memory. In this case, it is necessary to search for a contiguous storage space that can meet the ms requirement starting from the base address of the memory pool. When a contiguous storage space is found, a memory block is allocated for the data parameters. After allocating the memory block, the starting allocation address is not updated. Figure 5 In the diagram, the thick line segment starting from the beginning of the video memory indicates that memory space has been allocated.

[0072] The advantage of this setting is that it ensures the continuity of memory allocation in the memory pool as much as possible, avoiding excessive memory fragmentation.

[0073] Figure 6 This diagram illustrates an alternative memory block allocation method, intended to demonstrate how the invention enables the continuity of memory allocation within a memory pool through an example. Figure 6 As shown, A, B, C, and D are data parameters, where C = convolution(A, B); D = relu(C). A and B are released after the convolution kernel function `convolution` in the neural network is completed, while C, as the output of the convolution operation, will be used by the subsequent `relu` operation. Therefore, after A, B, and C are allocated, the initial allocation address will be updated to the end address of the memory block containing the data parameter C. Figure 6 As shown in the second line, after A and B are freed, a memory block needs to be allocated for data parameter D. If the memory block for D is allocated according to the starting allocation address, then C and D are contiguous memory blocks, as shown in the second line. Figure 6 As shown in the third line, if the memory block of D is searched directly from the base address of the memory pool, memory fragmentation will occur. Figure 6 This is for illustrative purposes only.

[0074] S130. When starting the device kernel function, a device kernel function start request is sent to the acceleration device in the AI ​​computing system, and the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform memory access operations according to the virtual address of each data parameter during the execution of the device kernel function.

[0075] Optionally, the AI ​​computing system may include a CPU and an acceleration device. The CPU can be used to execute the main control program, i.e., the AI ​​program, which includes AI computing tasks. The acceleration device is used to perform AI computing by executing kernel functions.

[0076] Optionally, a kernel function can refer to a computation function that executes on an accelerated device, typically containing highly parallel computational logic.

[0077] Optionally, the AI ​​program includes a kernel function startup request. When the CPU executes the kernel function startup request, the acceleration device executes the kernel function. In the prior art, the kernel function can be in the form of kernel(arg0, arg1, ...), where arg0 and arg1 can refer to different data parameters. After the CPU executes the kernel function startup instruction, it can directly pass the address pointer of the data parameters into the kernel function so that the acceleration device can perform read and write operations on the data parameters according to the passed address pointer.

[0078] Understandably, in existing technologies, storage space for data parameters is allocated directly in the device's global storage. The MMU in the acceleration device needs to map virtual addresses to physical addresses in the global device storage based on TLB or page tables. Therefore, the CPU only needs to pass the virtual address of the data parameters into the kernel function.

[0079] This invention aims to simplify existing address mapping schemes in AI application execution scenarios. Therefore, during the AI ​​program compilation phase, the entry kernel function on the acceleration device is transformed and replaced with a corresponding stub function. The stub function maps the global address pointer passed from the CPU to a virtual address in the acceleration device's address space before calling the kernel function. At this time, the MMU only needs to perform mapping based on the base address of the memory pool passed from the CPU and the virtual addresses of each data parameter in the acceleration device's address space. Through simple address offset operations, the virtual addresses of each data parameter can be mapped to the effective physical address space, thereby avoiding the lookup overhead of traditional TLB or page tables and improving the efficiency of address mapping.

[0080] The technical solution of this invention allocates a memory pool for the AI ​​program in the global storage of the acceleration device and executes the AI ​​program. This allows for pre-allocation of the memory pool before task startup, enabling the acceleration device to reuse the memory during operation. This avoids requesting memory from the CPU for each calculation, ensuring the AI ​​program's long-term exclusive access to storage. By allocating memory blocks for data parameters in the memory pool according to a circular linked list during memory allocation instruction execution, the continuity of memory allocation in the memory pool can be guaranteed as much as possible, avoiding excessive memory fragmentation and improving the performance of shared memory allocation. When starting the device kernel function, a device kernel function startup request is sent to the acceleration device in the AI ​​computing system, and the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function. This allows the acceleration device to obtain the virtual address of each data parameter in the acceleration device's address space by executing the stub function, and to perform memory access operations based on the virtual address of each data parameter during the execution of the device kernel function. This reduces the clock cycles consumed by the MMU in the acceleration device for address mapping. By replacing TLB and page table lookups with simple address mapping calculations, the efficiency of data parameter read and write operations is improved, thereby improving AI computing performance.

[0081] Example 2

[0082] Figure 7 This is a flowchart illustrating a storage management method for an acceleration device according to Embodiment 2 of the present invention. Based on the above embodiments, this embodiment specifically describes the storage management method for the acceleration device. Figure 7 As shown, the method includes:

[0083] S210. Allocate a memory pool for the AI ​​program in the global storage of the acceleration device and execute the AI ​​program.

[0084] S220. According to the memory allocation instruction, obtain the memory size required by the data parameters, and according to the circular linked list, obtain the starting allocation address of the memory pool.

[0085] S230. Determine whether the size of the contiguous storage space in the memory pool starting from the initial allocation address is greater than or equal to the memory size required by the data parameters; if yes, proceed to step S240; if no, proceed to step S250.

[0086] S240. Based on the starting allocation address and the memory size required by the data parameters, allocate a memory block for the data parameters in the memory pool and update the starting allocation address; proceed to step S260.

[0087] The process of allocating a memory block for the data parameters in the memory pool based on the starting allocation address and the memory size required by the data parameters, and updating the starting allocation address, may include:

[0088] Based on the initial allocation address and the memory size required by the data parameters, the boundary address of the memory block is determined in the memory pool, and the boundary address and occupancy status of the memory block are recorded in the circular linked list.

[0089] Update the starting allocation address based on the end boundary address of the currently allocated memory block.

[0090] Optionally, the boundary address of the memory block may include the start boundary address and the end boundary address of the memory block, and the occupancy status of the memory block may include occupied and free. After allocating a memory block in the memory pool, the occupancy status of the memory block can be recorded as occupied.

[0091] Optionally, the next byte position of the end boundary address of the currently allocated memory block can be used as the updated starting allocation address. For example, if the memory block allocated for data parameter A is 0x400-0x800, then the starting allocation address can be updated to 0x801. This is only an example.

[0092] S250. Select the target free memory block according to the circular linked list, and allocate a memory block for the data parameter in the memory pool according to the target free memory block and the memory size required by the data parameter; execute step S260.

[0093] S260. When starting the device kernel function, a device kernel function start request is sent to the acceleration device in the AI ​​computing system, and the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform memory access operations according to the virtual address of each data parameter during the execution of the device kernel function.

[0094] Optionally, the global address pointer of the data parameter can be the starting boundary address of the memory block corresponding to the data parameter. The circular linked list manages the boundary address and occupancy status of each memory block. In the AI ​​program, only the global address pointer of the data parameter can be passed. During the program compilation process, the memory size required by each data parameter is pre-compiled. Therefore, the obtained global address pointer of the data parameter can determine the address space where the data parameter is located in the memory pool.

[0095] Furthermore, methods for accelerating device storage management may also include:

[0096] During the execution of the memory release instruction, the memory block to be released is determined in the memory pool based on the global address pointer of the data parameter to be released and the circular linked list.

[0097] Perform a memory release operation on the memory block to be released, and change the memory block from occupied to free in the circular linked list.

[0098] Optionally, the AI ​​program executed on the CPU side has explicit memory allocation instructions and memory release instructions. The memory release instructions record the data parameters to be released. Based on the global address pointer of the data parameters to be released and the circular linked list, the memory block with the global address pointer of the data parameters to be released as the starting boundary address can be determined in the data managed by the circular linked list, and the memory release operation can be performed on the memory block.

[0099] Furthermore, after performing a memory release operation on the memory block to be released and changing the memory block's state from occupied to free in the circular linked list, it may also include:

[0100] Based on the circular linked list, perform any of the following:

[0101] If it is determined that the end boundary address of the currently released memory block is adjacent to the start allocation address, and there is no free memory block adjacent to the currently released memory block, then the start allocation address is updated according to the start boundary address of the currently released memory block.

[0102] If it is determined that the end boundary address of the currently released memory block is not adjacent to the start allocation address, and there is at least one free memory block adjacent to the currently released memory block, then the adjacent free memory block is merged with the currently released memory block.

[0103] If it is determined that the end boundary address of the currently released memory block is adjacent to the start allocation address, and there is a free memory block adjacent to the currently released memory block, then the adjacent free memory block is merged with the currently released memory block, and the start allocation address is updated according to the start boundary address of the free memory block.

[0104] The advantage of this setup is that memory can be reused through the release and merging of memory blocks, thus improving the allocation performance of shared memory.

[0105] The technical solution of this invention allocates a memory pool for the AI ​​program in the global storage of the acceleration device and executes the AI ​​program. This allows for pre-allocation of the memory pool before task startup, enabling the acceleration device to reuse the memory during operation. This avoids requesting memory from the CPU for each calculation, ensuring the AI ​​program's long-term exclusive access to storage. By allocating memory blocks for data parameters in the memory pool according to a circular linked list during memory allocation instruction execution, the continuity of memory allocation in the memory pool can be guaranteed as much as possible, avoiding excessive memory fragmentation and improving the performance of shared memory allocation. When starting the device kernel function, a device kernel function startup request is sent to the acceleration device in the AI ​​computing system, and the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function. This allows the acceleration device to obtain the virtual address of each data parameter in the acceleration device's address space by executing the stub function, and to perform memory access operations based on the virtual address of each data parameter during the execution of the device kernel function. This reduces the clock cycles consumed by the MMU in the acceleration device for address mapping. By replacing TLB and page table lookups with simple address mapping calculations, the efficiency of data parameter read and write operations is improved, thereby improving AI computing performance.

[0106] Example 3

[0107] Figure 8 This is a flowchart of a memory mapping method for an acceleration device according to Embodiment 3 of the present invention. This embodiment is applicable to situations where the acceleration device performs memory access operations on memory access instructions. This method can be executed by a memory mapping device of the acceleration device, which can be implemented in hardware and / or software, and is generally configured in an acceleration device with data processing capabilities. Figure 8 As shown, the method includes:

[0108] S310, in response to the CPU's request to start the device kernel function, triggers the execution of the stub function.

[0109] During the compilation of the executable file, based on the function prototype of the entry kernel function, a stub function corresponding to the parameters is generated to replace the device kernel function as the program entry point, and the stub function generates a call to the original kernel function.

[0110] Optionally, a device kernel function can refer to parallel computation running on an accelerated device. It typically contains code logic that is executed in large-scale parallelism. After the CPU sends a device kernel function start request to the accelerated device in the AI ​​computing system, it can pass the base address of the memory pool and the global address pointers of each data parameter in the device kernel function into the stub function.

[0111] Optionally, during program compilation, the compiler replaces the entry point of the device kernel function with a stub function, so that the program entry point is the stub function, and calls the device kernel function after executing the address translation and other logic of the stub function.

[0112] Optionally, during the compilation of the executable file, the process of using stub functions to replace device kernel functions may specifically include: performing front-end compilation on the source code of the device kernel function to obtain the intermediate representation of the device kernel function, generating the intermediate representation of the stub function, linking the intermediate representation of the device kernel function with the intermediate representation of the stub function to obtain the intermediate representation of the kernel function that incorporates the stub function, and then performing back-end compilation on the intermediate representation of the kernel function that incorporates the stub function to generate the binary file.

[0113] In an optional example, the original device function could be: kernel(arg0, arg1, ...); however, in this invention, the compiler can generate multiple device functions, including: stub(config, arg0, arg1, ...); {karg0 = convert_ptr(arg0); kernel(karg0, krag1, ...);}; where kernel(karg0, krag1, ...) can be called as a non-entry device function, and stub(config, arg0, arg1, ...) can be used as the entry function of the device and call kernel. The stub function receives the memory pool base address and global address pointers of each data parameter from the CPU. config can contain the memory pool base address in the device's global storage, arg0, arg1, etc., are global address pointers of different data parameters, and krag0 = convert_ptr(arg0) can be used to convert the global address pointers of the data parameters into virtual addresses in the accelerated device's address space.

[0114] S320: Based on the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the memory management unit, the pointer parameters of each data parameter are modified to obtain the virtual address of each data parameter in the address space of the acceleration device.

[0115] Optionally, the stub function modifies the pointer parameters of each data parameter during execution based on the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the storage management unit, in order to obtain the virtual address of each data parameter in the address space of the accelerated device; wherein, the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the accelerated device described in any embodiment of the present invention and are passed into the stub function when the device program starts; the address offset in the storage management unit is pre-configured during the compilation process of the executable file.

[0116] It is understandable that when an accelerated device performs computational tasks, these tasks are actually executed in parallel by multiple processor units. Therefore, each processor unit needs to map the global device storage into its local address space to perform the computation, such as... Figure 3 As shown, the device address space seen by the CPU is different from the device address space seen by the acceleration device. Therefore, after the global address pointer of the memory pool base address and data parameters is passed in, it is necessary to convert the address range of the global device storage into the address range that the processor unit can access locally to ensure that each processor unit can correctly address the required data.

[0117] Optionally, the address offset in the memory management unit can be pre-configured. The memory management unit is located within the acceleration device. The address offset can refer to the difference between the base address of the memory pool in the address range accessible by the CPU and the base address of the memory pool in the address range accessible by the processor unit.

[0118] An optional example illustrates how to map global address pointers of data parameters to virtual addresses of each data parameter in the address space of the accelerated device. For example:

[0119] On the CPU side, a memory pool is partitioned, with a base address of 0x400. Simultaneously, the global address pointers for data parameters arg0 and arg1 are 0x1500 and 0x3500 respectively. On the processor unit side, the base address of the memory pool is 0x100, resulting in an address offset of 0x300. Further, after the memory pool base address and the global address pointers of each data parameter are passed to the stub function, the pointer parameters of each data parameter are corrected according to karg0 = convert_ptr(arg0). The corrected karg0 = arg0 - 0x300, karg1 = arg1 - 0x300. At this point, karg0 = 0x1200, karg1 = 0x3200. That is, the virtual addresses of the data parameters in the acceleration device's address space are obtained. The kernel function call in the stub function actually passes the virtual addresses to the kernel function for further AI calculations.

[0120] Optionally, before the correction, the pointer parameter of the data parameter is a global address pointer; after the correction, the pointer parameter of the data parameter is a virtual address in the address space of the acceleration device.

[0121] The base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the acceleration device described in any one of Embodiments 1 and 2 of the present invention and passed into the stub function; the address offset in the storage management unit is pre-configured during the compilation of the executable file.

[0122] S330. During the execution of the device kernel function, the memory management unit is called to perform virtual and physical address mapping on each accessed data in the memory access instruction according to the address offset and the virtual address of each data parameter, so as to obtain the physical address of each accessed data in the global memory, and to perform the memory access operation according to the physical address of each accessed data.

[0123] Optionally, during the execution of the device kernel function, the storage management unit performs virtual-physical address mapping on the data addresses involved in the memory access instruction based on the address offset and the virtual addresses of each data parameter, so as to obtain the physical address of the data involved in the memory access instruction in the global storage, and performs the memory access operation according to the physical address of each accessed data.

[0124] Optionally, the storage management unit proposed in this invention has been greatly simplified compared to existing storage management units. In this invention, the storage management unit does not need to perform TLB or page table lookups. The storage management unit only needs to add the virtual address of the accessed data to the address offset to obtain the physical address of the accessed data in the global storage.

[0125] Understandably, data parameters can refer to data objects defined in a program that need to be processed. In heterogeneous computing, data parameters are logical concepts, representing data entities that need to be stored and accessed in memory. These can be data such as arrays or tensors. Accessed data, on the other hand, can refer to the memory data actually operated on by specific memory access instructions when the kernel function is executed. This is a physical concept, representing the specific bit stream read from or written to memory at the moment the instruction is executed. For example, in the instruction LOAD R1,[0x20010000], 0x20010000 points to the memory data.

[0126] The technical solution of this invention, in response to the CPU's request to start the device kernel function, triggers the execution of a stub function. Based on the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the memory management unit, the pointer parameters of each data parameter are corrected to obtain the virtual address of each data parameter in the accelerated device address space. This method allows the stub function to replace the device kernel function, quickly correcting the global address pointer to the virtual address in the accelerated device address space. Furthermore, during the execution of the device kernel function, the memory management unit is called to perform virtual-physical address mapping on each accessed data in the memory access instruction based on the address offset and the virtual address of each data parameter, obtaining the physical address of each accessed data in the global memory. The memory access operation is then performed based on the physical address of each accessed data. This method enables the memory management unit to obtain the physical address of each accessed data in the global memory through a simplified mapping method, solving the problem of the extra clock cycles consumed by the memory management unit to perform TLB or page table lookups in existing technologies.

[0127] Example 4

[0128] Figure 9 This is a schematic diagram of the storage management device for an acceleration device provided in Embodiment 4 of the present invention. Figure 9 As shown, the device includes: a memory pool allocation module 410, a memory block allocation module 420, and a device kernel function startup module 430.

[0129] The memory pool allocation module 410 is used to allocate a memory pool for the AI ​​program in the global storage of the acceleration device and execute the AI ​​program; wherein the memory pool is managed in the form of a circular linked list.

[0130] The memory block allocation module 420 is used to allocate memory blocks for data parameters in the memory pool according to the circular linked list during the execution of memory allocation instructions; wherein the memory blocks are represented by global address pointers.

[0131] The device kernel function startup module 430 is used to send a device kernel function startup request to the acceleration device in the AI ​​computing system during the execution of the startup instruction for the device kernel function, and pass the base address of the memory pool and the global address pointers of each data parameter in the device kernel function into the stub function, so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform data parameter memory access operations according to the virtual address of each data parameter during the execution of the device kernel function.

[0132] The technical solution of this invention allocates a memory pool for the AI ​​program in the global storage of the acceleration device and executes the AI ​​program. This allows for pre-allocation of the memory pool before task startup, enabling the acceleration device to reuse the memory during operation. This avoids requesting memory from the CPU for each calculation, ensuring the AI ​​program's long-term exclusive access to storage. By allocating memory blocks for data parameters in the memory pool according to a circular linked list during memory allocation instruction execution, the continuity of memory allocation in the memory pool can be guaranteed as much as possible, avoiding excessive memory fragmentation and improving the performance of shared memory allocation. When starting the device kernel function, a device kernel function startup request is sent to the acceleration device in the AI ​​computing system, and the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function. This allows the acceleration device to obtain the virtual address of each data parameter in the acceleration device's address space by executing the stub function, and to perform memory access operations based on the virtual address of each data parameter during the execution of the device kernel function. This reduces the clock cycles consumed by the MMU in the acceleration device for address mapping. By replacing TLB and page table lookups with simple address mapping calculations, the efficiency of data parameter read and write operations is improved, thereby improving AI computing performance.

[0133] Based on the above embodiments, the memory block allocation module 420 can be specifically used for:

[0134] According to the memory allocation instruction, obtain the memory size required for the data parameters, and according to the circular linked list, obtain the starting allocation address of the memory pool;

[0135] Determine whether the size of the contiguous storage space in the memory pool, starting from the initial allocation address, is greater than or equal to the memory size required by the data parameters;

[0136] If so, then according to the starting allocation address and the memory size required by the data parameters, allocate a memory block for the data parameters in the memory pool and update the starting allocation address;

[0137] If not, then select the target free memory block according to the circular linked list, and allocate a memory block for the data parameters in the memory pool according to the target free memory block and the memory size required by the data parameters.

[0138] Based on the above embodiments, the memory block allocation module 420 can be further specifically used for:

[0139] Based on the initial allocation address and the memory size required by the data parameters, the boundary address of the memory block is determined in the memory pool, and the boundary address and occupancy status of the memory block are recorded in the circular linked list.

[0140] Update the starting allocation address based on the end boundary address of the currently allocated memory block.

[0141] Based on the above embodiments, a memory block release module may also be included, used for:

[0142] During the execution of the memory release instruction, the memory block to be released is determined in the memory pool based on the global address pointer of the data parameter to be released and the circular linked list.

[0143] Perform a memory release operation on the memory block to be released, and change the memory block from occupied to free in the circular linked list.

[0144] Based on the above embodiments, a memory block merging module may also be included, for:

[0145] Based on the circular linked list, perform any of the following:

[0146] If it is determined that the end boundary address of the currently released memory block is adjacent to the start allocation address, and there is no free memory block adjacent to the currently released memory block, then the start allocation address is updated according to the start boundary address of the currently released memory block.

[0147] If it is determined that the end boundary address of the currently released memory block is not adjacent to the start allocation address, and there is at least one free memory block adjacent to the currently released memory block, then the adjacent free memory block is merged with the currently released memory block.

[0148] If it is determined that the end boundary address of the currently released memory block is adjacent to the start allocation address, and there is a free memory block adjacent to the currently released memory block, then the adjacent free memory block is merged with the currently released memory block, and the start allocation address is updated according to the start boundary address of the free memory block.

[0149] The storage management device for the acceleration device provided in this embodiment of the invention can execute the storage management method for the acceleration device provided in any one of the embodiments of the invention (1 and 2), and has the corresponding functional modules and beneficial effects of the method.

[0150] Example 5

[0151] Figure 10 This is a schematic diagram of the structure of a storage mapping device for an acceleration device provided in Embodiment 5 of the present invention. Figure 10 As shown, the device includes: a stub function startup module 510, a pointer parameter correction module 520, and an address mapping module 530.

[0152] The stub function startup module 510 is used to trigger the execution of the stub function in response to the CPU's startup request for the device kernel function; wherein, during the compilation process of the executable file, the stub function is used in advance to replace the device kernel function as the program entry point, and the stub function generates the call to the device kernel function.

[0153] The pointer parameter correction module 520 is used to correct the pointer parameters of each data parameter according to the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the storage management unit, so as to obtain the virtual address of each data parameter in the address space of the acceleration device; wherein, the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the acceleration device as described in any one of Embodiments 1 and 2 of the present invention and passed into the stub function; the address offset in the storage management unit is pre-configured during the compilation process of the executable file.

[0154] The address mapping module 530 is used to call the storage management unit to perform virtual and physical address mapping on each accessed data in the memory access instruction according to the address offset and the virtual address of each data parameter during the execution of the device kernel function, so as to obtain the physical address of each accessed data in the memory access instruction in the global storage, and perform memory access operation according to the physical address of each accessed data.

[0155] The technical solution of this invention, in response to the CPU's request to start the device kernel function, triggers the execution of a stub function. Based on the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the memory management unit, the pointer parameters of each data parameter are corrected to obtain the virtual address of each data parameter in the accelerated device address space. This method allows the stub function to replace the device kernel function, quickly correcting the global address pointer to the virtual address in the accelerated device address space. Furthermore, during the execution of the device kernel function, the memory management unit is called to perform virtual-physical address mapping on each accessed data in the memory access instruction based on the address offset and the virtual address of each data parameter, obtaining the physical address of each accessed data in the global memory. The memory access operation is then performed based on the physical address of each accessed data. This method enables the memory management unit to obtain the physical address of each accessed data in the global memory through a simplified mapping method, solving the problem of the extra clock cycles consumed by the memory management unit to perform TLB or page table lookups in the prior art.

[0156] The storage mapping device for the acceleration device provided in this embodiment of the invention can execute the storage mapping method for the acceleration device provided in Embodiment 3 of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0157] Example 6

[0158] Figure 11A schematic diagram of an AI computing system 10, which can be used to implement embodiments of the present invention, is shown. The AI ​​computing system is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The AI ​​computing system can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0159] like Figure 11 As shown, the AI ​​computing system 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the AI ​​computing system 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0160] Multiple components in the AI ​​computing system 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the AI ​​computing system 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0161] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the memory management method and memory mapping method of the acceleration device as described in the embodiments of the present invention.

[0162] In some embodiments, the storage management method and storage mapping method of the acceleration device can be implemented as computer programs tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the AI ​​computing system 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the storage management method and storage mapping method of the acceleration device described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the storage management method and storage mapping method of the acceleration device by any other suitable means (e.g., by means of firmware).

[0163] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0166] To provide interaction with the user, the systems and techniques described herein can be implemented on an AI computing system having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the AI ​​computing system. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0167] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0168] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0169] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0170] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A storage management method for an acceleration device, characterized in that, Executed by the central processing unit (CPU) in an artificial intelligence (AI) computing system, including: In the global storage of the acceleration device, a memory pool is allocated for the AI ​​program, and the AI ​​program is executed; wherein, the memory pool is managed in the form of a circular linked list; During the execution of memory allocation instructions, memory blocks are allocated for data parameters in the memory pool according to the circular linked list; wherein, the memory blocks are represented by global address pointers; When starting the device kernel function, a device kernel function start request is sent to the acceleration device in the AI ​​computing system. The base address of the memory pool and the global address pointers of each data parameter in the device kernel function are passed to the stub function so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform memory access operations based on the virtual address of each data parameter during the execution of the device kernel function.

2. The method according to claim 1, characterized in that, Based on the circular linked list, memory blocks are allocated for the data parameters in the memory pool, including: According to the memory allocation instruction, obtain the memory size required for the data parameters, and according to the circular linked list, obtain the starting allocation address of the memory pool; Determine whether the size of the contiguous storage space in the memory pool, starting from the initial allocation address, is greater than or equal to the memory size required by the data parameters; If so, then according to the starting allocation address and the memory size required by the data parameters, allocate a memory block for the data parameters in the memory pool and update the starting allocation address; If not, then select the target free memory block according to the circular linked list, and allocate a memory block for the data parameters in the memory pool according to the target free memory block and the memory size required by the data parameters.

3. The method according to claim 2, characterized in that, Based on the starting allocation address and the memory size required by the data parameters, allocate a memory block for the data parameters in the memory pool and update the starting allocation address, including: Based on the initial allocation address and the memory size required by the data parameters, the boundary address of the memory block is determined in the memory pool, and the boundary address and occupancy status of the memory block are recorded in the circular linked list. Update the starting allocation address based on the end boundary address of the currently allocated memory block.

4. The method according to claim 1, characterized in that, Also includes: During the execution of the memory release instruction, the memory block to be released is determined in the memory pool based on the global address pointer of the data parameter to be released and the circular linked list. Perform a memory release operation on the memory block to be released, and change the memory block from occupied to free in the circular linked list.

5. The method according to claim 4, characterized in that, After performing a memory release operation on the memory block to be released and changing the memory block's state from occupied to free in the circular linked list, the process also includes: Based on the circular linked list, perform any of the following: If it is determined that the end boundary address of the currently released memory block is adjacent to the start allocation address, and there is no free memory block adjacent to the currently released memory block, then the start allocation address is updated according to the start boundary address of the currently released memory block. If it is determined that the end boundary address of the currently released memory block is not adjacent to the start allocation address, and there is at least one free memory block adjacent to the currently released memory block, then the adjacent free memory block is merged with the currently released memory block. If it is determined that the end boundary address of the currently released memory block is adjacent to the start allocation address, and there is a free memory block adjacent to the currently released memory block, then the adjacent free memory block is merged with the currently released memory block, and the start allocation address is updated according to the start boundary address of the free memory block.

6. A storage mapping method for an acceleration device, characterized in that, Performed by accelerated devices in artificial intelligence (AI) computing systems, including: In response to the CPU's request to start the device kernel function, the execution of the stub function is triggered; wherein, during the compilation of the executable file, the stub function is used in advance to replace the device kernel function as the program entry point, and the call to the device kernel function is generated in the stub function; Based on the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the storage management unit, the pointer parameters of each data parameter are modified to obtain the virtual address of each data parameter in the address space of the accelerated device; wherein, the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the accelerated device according to any one of claims 1-5 of this invention and passed into the stub function; the address offset in the storage management unit is pre-configured during the compilation process of the executable file; During the execution of the device kernel function, the memory management unit is invoked to perform virtual and physical address mapping on each data accessed in the memory access instruction based on the address offset and the virtual address of each data parameter, so as to obtain the physical address of each data accessed in the memory access instruction in the global storage, and to perform the memory access operation based on the physical address of each data accessed.

7. A storage management device for an acceleration device, characterized in that, Executed by the central processing unit (CPU) in an artificial intelligence (AI) computing system, including: The memory pool allocation module is used to allocate a memory pool for the AI ​​program in the global storage of the acceleration device and execute the AI ​​program; wherein the memory pool is managed in the form of a circular linked list; The memory block allocation module is used to allocate memory blocks for data parameters in the memory pool according to the circular linked list during the execution of memory allocation instructions; wherein, the memory blocks are represented by global address pointers; The device kernel function startup module is used to send a device kernel function startup request to the acceleration device in the AI ​​computing system during the execution of the startup instruction for the device kernel function. It also passes the base address of the memory pool and the global address pointers of each data parameter in the device kernel function to the stub function, so that the acceleration device can obtain the virtual address of each data parameter in the address space of the acceleration device by executing the stub function, and perform data parameter memory access operations based on the virtual address of each data parameter during the execution of the device kernel function.

8. A storage mapping device for an acceleration device, characterized in that, Performed by accelerated devices in artificial intelligence (AI) computing systems, including: The stub function startup module is used to respond to the CPU's startup request for the device kernel function and trigger the execution of the stub function; wherein, during the compilation process of the executable file, the stub function is used in advance to replace the device kernel function as the program entry point, and the stub function generates the call to the device kernel function; The pointer parameter correction module is used to correct the pointer parameters of each data parameter according to the base address of the memory pool, the global address pointers of each data parameter in the device kernel function, and the address offset in the storage management unit, so as to obtain the virtual address of each data parameter in the address space of the acceleration device; wherein, the base address of the memory pool and the global address pointers of each data parameter in the device kernel function are generated by the storage management method of the acceleration device according to any one of claims 1-5 of the present invention and passed into the stub function; the address offset in the storage management unit is pre-configured during the compilation process of the executable file; The address mapping module is used to call the storage management unit to perform virtual and physical address mapping on each accessed data in the memory access instruction based on the address offset and the virtual address of each data parameter during the execution of the device kernel function. This is to obtain the physical address of each accessed data in the global storage and to perform the memory access operation based on the physical address of each accessed data.

9. An AI computing system, characterized in that, The AI ​​computing system includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the storage management method of the acceleration device according to any one of claims 1-5 and the storage mapping method of the acceleration device according to claim 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the storage management method of the acceleration device according to any one of claims 1-5 and the storage mapping method of the acceleration device according to claim 6.

Images