Memory management method and apparatus

Abstract

A memory management method and apparatus, which are used for reducing memory fragments when a computing device runs a program. In the present application, the method comprises: determining a data type of first data to be stored; when the data type of the first data is a long life cycle type, selecting, from a first memory pool, a first memory block having the same size as the first data to store the first data; and when the data type of the first data is a short life cycle type, selecting, from a second memory pool, a second memory block to store the first data, wherein the size of the second memory block is greater than or equal to the size of the first data. The first memory pool and the second memory pool record memory blocks which are applied from a memory but have not been used, and the memory blocks correspond to physical spaces in the memory.

Description

Memory management method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on March 28, 2024, with application number 202410372510.5 and invention name "A Memory Management Method and Device", the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of computers, and in particular to a memory management method and device. Background Art

[0004] When a computing device runs a program, it generates data (such as tensors). Based on the size of the data, the computing device allocates physical space from memory and then maps the physical space into logical memory blocks, which are used to store the data. When the data stored in a memory block is released, the memory block can be used to store other data.

[0005] However, when a memory block is larger than the size of the data currently stored, there will be some unused space in the block, which can potentially lead to memory fragmentation. For example, if three memory blocks have 20MB, 30MB, and 50MB of unused space respectively, and a new data block of 90MB needs to be stored, although the three unused spaces add up to 100MB, none of them can store the 90MB of data. Moreover, if the data needs to occupy these memory blocks for a long time, the unused space will lead to memory fragmentation.

[0006] How to reduce memory fragmentation is a technical problem that needs to be solved urgently. Summary of the Invention

[0007] The present application provides a memory management method and apparatus for reducing memory fragmentation during program execution on a computing device.

[0008] In a first aspect, the present application provides a memory management method, which can be specifically executed by a computing device, or by a main processor (eg, a central processing unit (CPU)) in the computing device.

[0009] For ease of description, the following takes CPU execution as an example.

[0010] The memory management method includes: a CPU determining the data type of first data to be stored. If the data type of the first data is a long-life type, the CPU selects a first memory block of the same size as the first data from a first memory pool to store the first data; if the data type of the first data is a short-life type, the CPU selects a second memory block from a second memory pool to store the first data, where the size of the second memory block is greater than or equal to the size of the first data. The first and second memory pools record unused memory blocks requested by the CPU from memory, and the memory blocks correspond to physical space in memory.

[0011] In the above technical solution, because long-lived data occupies memory blocks for a long time, the CPU selects a memory block for the long-lived data from the first memory pool, and the size of the selected first memory block is equal to the size of the first data. This leaves no remaining space in the first memory block when storing the first data, helping to reduce memory fragmentation when the computing device runs programs.

[0012] In one possible implementation, if the data type of the first data is a long-lived type and the CPU fails to select a first memory block in the first memory pool, the CPU requests a third memory block of the same size as the first data from the main memory to store the first data. After the first data releases the third memory block, the CPU places the third memory block into the first memory pool for management.

[0013] In the above technical solution, when the data type of the first data is a long-life type, if the CPU determines that the selection of the first memory block in the first memory pool has failed, it can further apply for a third memory block of the same size as the first data from the memory. In this way, the third memory block has no remaining space when storing the first data, which helps to reduce the formation of memory fragmentation during the operation of the computing device program. Furthermore, the CPU places the third memory block released by the first data into the first memory pool for management, so that when there is new data to be stored that is the same size as the first data, the CPU can also select a third memory block from the first memory pool and store the new data to be stored in the third memory block.

[0014] In one possible implementation, when the data type of the first data is a long-life type and the CPU fails to apply for a third memory block from the memory, the CPU selects a fourth memory block from the second memory pool to store the first data, and the size of the fourth memory block is greater than or equal to the size of the first data; after the first data releases the fourth memory block, the CPU puts the fourth memory block into the second memory pool for management.

[0015] In the above technical solution, when the data type of the first data is a long-life type, if the CPU determines that the request for the third memory block from the main memory has failed, it can select a memory block from the second memory pool, which helps to find the memory block for storing the first data. Furthermore, the CPU returns the fourth memory block released by the first data to the second memory pool for management, so that when new data of a short-life type is required to be stored, the CPU can also select a fourth memory block from the second memory pool and store the new data in the fourth memory block.

[0016] In one possible implementation, when the data type of the first data is a long-life type and the CPU fails to select the fourth memory block from the second memory pool, the CPU releases the first memory pool and the second memory pool to the memory; the CPU re-applies for a fifth memory block of the same size as the first data from the memory to store the first data.

[0017] In the above technical solution, when the data type of the first data is a long-life cycle type, if the CPU determines that the selection of the fourth memory block from the second memory pool has failed, the current memory pool can be released to increase the free physical space in the memory (i.e., physical memory), and the CPU can then reapply for the fifth memory block from the memory, which helps to improve the success rate of the CPU in determining the memory block for storing the first data.

[0018] In one possible implementation, the first data is data generated during the operation of an artificial intelligence (AI) model, and the memory is the memory of an AI accelerator card. The AI ​​accelerator card is, for example, a graphics processing unit (GPU) or a neural-network processing unit (NPU).

[0019] In the above technical solution, specific applicable scenarios of this solution are provided, which helps to reduce the formation of memory fragmentation during the operation of AI models.

[0020] In one possible implementation, when the data type of the first data is a long life cycle type, the first data is data generated but not released in any round of iteration of the AI ​​model operation, data generated but not released in the forward operation of any round of iteration of the AI ​​model operation, data generated in the initialization of the AI ​​model, or data generated in the initialization of the optimizer.

[0021] In the above technical solution, providing a representation form of the first data of the long life cycle type helps the CPU to accurately and quickly identify the first data of the long life cycle type.

[0022] In a possible implementation, after the optimizer is initialized, the CPU releases the first memory pool and the second memory pool into the main memory.

[0023] In the above technical solution, it is taken into account that the data generated during the initialization of the AI ​​model and the optimizer may include data of short life cycle type and data of long life cycle type, and these data will be considered by the CPU as data of long life cycle type; and after the optimizer is initialized, the data of short life cycle type generated during the initialization of the AI ​​model and the optimizer will release the memory blocks previously occupied by them to the first memory pool. In this way, the CPU can further release the first memory pool to the memory, which helps to reduce memory fragmentation and reduce the degree of memory tension. In addition, the CPU also releases the second memory pool to the memory, which helps to further reduce memory fragmentation and reduce the degree of memory tension.

[0024] In one possible implementation, when the data type of the first data is a short-lifecycle type and the CPU fails to select a second memory block from the second memory pool, the CPU determines that the first data is not data generated in the forward operation of any iterative round of the AI ​​model execution, and then selects a memory block from the first memory pool to store the first data.

[0025] In the above technical solution, since data of long life cycle type may include data generated but not released in the forward operation of each iteration of the AI ​​model, this data may occupy the memory blocks in the first memory pool during part of the time period of a round of iteration (for example, the gradient calculation generated in the forward operation to the reverse operation), so the memory allocator may allow data of short life cycle type generated in other time periods other than the forward operation (such as the reverse operation period, the optimizer initialization period, etc.) to occupy the memory blocks in the first memory pool. In this way, when data of short life cycle type occupies the memory blocks in the first memory pool, it helps to reduce its impact on the storage of data of long life cycle type.

[0026] In one possible implementation, the first data is any one of X first data generated in the Nth iteration of the AI ​​model operation, where X is an integer greater than or equal to 1, and N is an integer greater than or equal to 1. Before determining the data type of the first data to be stored, the CPU may further determine the generation order of the long-lifecycle type second data among the X second data, where the X second data are data generated in the Mth iteration of the AI ​​model operation, and M is an integer greater than or equal to 1 and less than N. Accordingly, when determining the data type of the first data to be stored, the CPU may specifically determine the data type of the first data based on the generation order of the first data among the X first data and the generation order of the long-lifecycle type second data among the X second data.

[0027] In the above technical solution, considering that the data classification generated in each iteration is the same, for example, in the Mth iteration, weight 1, weight 2, and activation value 1 are generated in sequence, then in the Nth iteration, weight 1, weight 2, and activation value 1 are generated in the same sequence. The difference is that the values ​​of weight 1, weight 2, and activation value 1 in these two iterations are different. In this way, the generation order of long-lifecycle type data among the X data is determined in the Mth iteration, and this generation order is then applied to the subsequent data type judgment.

[0028] In a second aspect, the present application provides a memory management device, which may specifically be a computing device, or a main processor (eg, a CPU) in the computing device.

[0029] The memory management device may include a determination module and a memory block management module.

[0030] The determination module is used to determine the data type of the first data to be stored.

[0031] The memory block management module is configured to select a first memory block of the same size as the first data from a first memory pool to store the first data when the data type of the first data is long-lifecycle type; and to select a second memory block from a second memory pool to store the first data when the data type of the first data is short-lifecycle type, where the size of the second memory block is greater than or equal to the size of the first data. The first memory pool and the second memory pool record unused memory blocks requested from the memory, and the memory blocks correspond to physical space in the memory.

[0032] In one possible implementation, when the data type of the first data is a long-life type and the selection of the first memory block in the first memory pool fails, the memory block management module is further used to apply for a third memory block of the same size as the first data from the memory to store the first data; and after the first data releases the third memory block, the third memory block is placed in the first memory pool for management.

[0033] In one possible implementation, when the data type of the first data is a long-life type and applying for a third memory block from the memory fails, the memory block management module is further used to select a fourth memory block from the second memory pool to store the first data, and the size of the fourth memory block is greater than or equal to the size of the first data; and after the first data releases the fourth memory block, the fourth memory block is placed in the second memory pool for management.

[0034] In one possible implementation, when the data type of the first data is a long-life type and the selection of the fourth memory block from the second memory pool fails, the memory block management module is also used to release the first memory pool and the second memory pool to the memory, and to apply for a fifth memory block of the same size as the first data from the memory again to store the first data.

[0035] In a possible implementation, after the optimizer is initialized, the memory block management module is further configured to release the first memory pool and the second memory pool into the memory.

[0036] In a third aspect, the present application provides a computing device comprising a main processor and an AI accelerator card, wherein the main processor is used to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect, and the AI ​​accelerator card is used to generate first data and second data to be stored.

[0037] In one possible implementation, the AI ​​accelerator card is used to generate first data to be stored. The main processor is used to determine the data type of the first data to be stored. When the data type of the first data is a long life cycle type, a first memory block of the same size as the first data is selected from the first memory pool, and the AI ​​accelerator card is instructed to store the first data in the first memory block; when the data type of the first data is a short life cycle type, the CPU selects a second memory block from the second memory pool, and the size of the second memory block is greater than or equal to the size of the first data, and the AI ​​accelerator card is instructed to store the first data in the second memory block. Among them, the first memory pool and the second memory pool record the memory blocks that have not been used by the main processor from the memory of the AI ​​accelerator card, and the memory blocks correspond to the physical space in the memory of the AI ​​accelerator card.

[0038] In a fourth aspect, the present application provides a computing device comprising a processor, the processor being connected to a memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the computing device executes the method of the above-mentioned first aspect or any possible implementation of the first aspect.

[0039] In a fifth aspect, the present application provides a chip comprising at least one processor and a memory; the memory is used to store computer programs; and at least one processor is used to execute the computer program to implement the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0040] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program or instructions. When the computer program or instructions are executed by a computing device, the method in the above-mentioned first aspect or any possible implementation of the first aspect is implemented.

[0041] In a seventh aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, it implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0042] The technical effects that can be achieved in any of the second to seventh aspects can refer to the description of the beneficial effects in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a schematic diagram of a computing architecture;

[0044] FIG2 is a schematic diagram of the structure of a neural network;

[0045] FIG3 is a schematic diagram of a forward operation and a reverse operation;

[0046] FIG4 is a schematic diagram of the architecture of a first computing device exemplarily provided in this application;

[0047] FIG5 is a schematic diagram of the architecture of a second computing device exemplarily provided in this application;

[0048] FIG6 is a schematic diagram of the architecture of a third computing device exemplarily provided in this application;

[0049] FIG7 is a schematic diagram showing the duration of data occupying a memory block during the operation of an AI model exemplified in this application;

[0050] FIG8 is a flow chart of a memory management method exemplarily provided in this application;

[0051] FIG9 is a flow chart of a method for determining a memory block exemplarily provided in the present application;

[0052] FIG10 is a flow chart of another memory management method exemplarily provided in this application;

[0053] FIG11 is a layered schematic diagram of a neural network framework exemplarily provided in this application;

[0054] FIG12 is a schematic structural diagram of a memory management device exemplarily provided in this application. DETAILED DESCRIPTION

[0055] The following first explains the relevant technical features involved in the embodiments of the present application. It should be noted that these explanations are intended to make the embodiments of the present application easier to understand and should not be regarded as limiting the scope of protection claimed by the present application.

[0056] 1. Block: A memory block obtained by the memory allocator through the underlying interface of the computing architecture. Blocks can also be called memory blocks. Memory blocks are logical concepts, or you can understand memory blocks as logical addresses. There is a mapping relationship between memory blocks and physical spaces in memory (for example, one or more physical pages), and this mapping relationship can be maintained in the underlying interface. The size of a memory block is determined when the memory allocator requests a memory block from the underlying interface. Furthermore, the memory allocator can request and release memory blocks from the underlying interface in units of memory blocks.

[0057] 2. Computing Architecture

[0058] Figure 1 is a schematic diagram of a computing architecture. This computing architecture can be a heterogeneous computing architecture, which includes the language interface layer, service layer, compilation layer, execution layer, and foundation layer from top to bottom. The explanation of each layer is as follows:

[0059] The language interface layer provides an application programming interface (API) library for device management, context management, stream management, memory management, model loading and execution, operator loading and execution, media data processing, and graph management, allowing users to call these APIs when developing artificial intelligence (AI) applications. For example, the API library may include operator development interfaces, model development interfaces, and application development interfaces.

[0060] The service layer provides operator libraries and computational optimization engines. Examples of operator libraries include neural network (NN) libraries and basic linear algebra subprograms (BLAS) libraries. Computational optimization engines include operator optimization, subgraph optimization, gradient optimization, model compression, and AI framework adapters.

[0061] The compilation layer provides support for operator development and includes the graph compiler and tensor boost engine (TBE) operators. The graph compiler compiles the computational graph of the user-entered intermediate representation (IR) into a model run by the neural-network processing unit (NPU); the TBE operator allows users to develop custom operators.

[0062] The execution layer is used for the execution of models and operators, and provides functional units such as the runtime library, graph executor, and artificial intelligence preprocessing (AIPP).

[0063] The basic layer is used to provide basic services, such as shared virtual memory (SVM), device virtualization (VM), host device communication (HDC), etc.

[0064] Furthermore, the upper layer of the computing architecture is the application layer, which includes AI frameworks such as TensorFlow and PyTorch. The lower layer of the computing architecture is the hardware layer, which may include computing resources such as central processing units (CPUs), graphics processing units (GPUs), and NPUs, and storage resources such as memory and hard drives.

[0065] 3. Tensor: A type of data generated during the operation of an AI model. The memory allocator needs to place tensors in memory blocks. Specifically, the memory allocator can only place one tensor continuously in a memory block, and one or more tensors can be placed in a memory block. When the size of the tensor placed in a memory block (or the amount of data) is smaller than the size of the memory block, the memory block can be considered split, where one part of the split is used to store the tensor, and the other part can be considered a new memory block, which can be used to store other tensors.

[0066] In addition, in addition to generating tensors during AI model operation, scalars can also be generated. The storage method of scalars is similar to that of tensors. This application can refer to tensors and scalars as data generated during AI model operation. The following description can mainly use tensors as an example.

[0067] 4. Memory fragmentation: When a memory block is larger than the size of the data currently stored, there is unused space in the block, which becomes a potential memory fragment. For example, if three memory blocks have 20MB, 30MB, and 50MB of unused space respectively, and a new data block of 90MB needs to be stored, even though the three unused spaces add up to 100MB, none of them can store the 90MB of data, nor can they be freed, resulting in memory fragmentation.

[0068] 5. Memory pool: a data structure used to maintain memory blocks. When the AI ​​model starts running, the memory pool is empty. When the memory allocator schedules newly generated data (such as tensors) to be stored in memory blocks, it may first select / search in the memory pool to determine whether a memory block can be found (or a part of the memory block can be split out) to store the data. If the memory allocator fails to find a memory block for storing the data from the memory pool, it will call the underlying interface to apply for a new memory block from the memory to store the data. When a piece of data is released, the memory block occupied by the data will become free, and the memory allocator will add the free memory block to the memory pool again. Furthermore, if the memory block is split, the memory allocator will try to merge the split memory blocks. For example, the memory allocator applies for memory block a through the underlying interface, and when storing data in memory block a, it splits memory block a into memory block a1 and memory block a2. If the memory allocator further determines that the memory pool includes memory block a1 and memory block a2, it will merge memory block a1 and memory block a2 into memory block a.

[0069] It is worth noting that the memory blocks stored in the memory pool are memory blocks in an idle state. When a memory block in the memory pool is used to store data, the memory allocator will remove the memory block from the memory pool; when the data stored in the memory block is released, the memory allocator will further add the memory block to the memory pool. Exemplarily, the memory pool can be considered as a data structure maintained by the memory allocator, which includes the identifier of the memory block. When the memory block is used to store data, the memory allocator can delete the identifier of the memory block in the data structure; when the data stored in the memory block is released, the memory allocator can add the identifier of the memory block back to the data structure.

[0070] 6. Neural networks (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Information processing can be achieved by adjusting the interconnected relationships between a large number of nodes within the neural network. Neural networks have the ability to self-learn and adapt. Specifically, a neural network usually contains multiple layers connected end to end, such as convolutional layers, fully connected layers (FC), activation layers, or pooling layers. Each layer can be expressed as a function y = f w(x), where f is the function, w is the weight, x is the input, and y is the output. Figure 2 is a schematic diagram of the structure of a neural network, which may include m layers connected end to end, where m is an integer greater than or equal to 2. Layer 0 of the neural network can be expressed as function f0, where f0's input is x, its output is y0, and its weight is w0; layer 1 of the neural network can be expressed as function f1, where f1's input is y0, its output is y1, and its weight is w1, and so on.

[0071] 7. AI model training: There is a data set {(x0,l0),…,(x n-1 ,l n-1 )}, where x0,…,x n-1 There are n inputs, and the corresponding l0,…,l n-1 are the expected outputs of these n inputs. j ,l j ) is called a sample data. j ) is input into the m-1 layer of the neural network as shown in Figure 2, and the output of the neural network can be obtained. The output of the neural network can be expressed as The goal of AI model training is to solve w0,…,w m-1 , so that under the loss function L, and l j The closest. For example, the solution process can use the forward operation (forward) and the backward operation (backward) shown in Figure 3. Among them, the forward operation: any input in the data set (which can be represented as x j ) is input to function f0, so that function f0 outputs Then, Input to function f1, so function f1 outputs By analogy, we can get functions f0 to f m-1 Corresponding to the output, namely Combined with x j The corresponding l j , and the loss function L to calculate the loss. Backward propagation: Use the chain rule to calculate each layer y in turn j The gradient Δy j , w j The gradient Δw j For example, through loss and y m-1 Determine the gradient Δy of the m-1th layer m-1 ; Then according to Δy m-1 and w m-1 Determine the gradient Δw of the m-1th layer m-1; And so on, we get Δy and Δw of each layer, that is, we get Δy0, Δw0, ..., Δy m-1 , Δw m-1 .

[0072] 8. Iteration (step): During AI model training, the training samples need to be repeatedly calculated and iterated many times. Each iteration is called a step. The operations performed in each iteration are almost the same.

[0073] In conjunction with the explanation of the above-mentioned related technical features, FIG4 is a schematic diagram of the architecture of the first computing device exemplarily provided in this application:

[0074] The computing device may include a processor, a memory, and a communication interface.

[0075] The processor can be used to read the AI ​​model code in the memory, run the AI ​​model according to the AI ​​model code, and write data generated during the AI ​​model running to the memory. Among them, AI model running can include AI model training and AI model use. AI model use can specifically be based on the trained AI model and AI model input to obtain AI model output.

[0076] The processor may specifically be a CPU, GPU, NPU, or an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on chip (SoC), or a complex programmable logic device (CPLD). The processor may be one or more. When there are multiple processors, the multiple processors may include processors of different types.

[0077] Storage may include memory and hard disk.

[0078] Memory refers to the internal storage that exchanges data directly with the processor. It can read and write data at any time and at a very high speed. It serves as a temporary data storage for the operating system or other running programs running on the processor. Memory includes volatile memory, such as random access memory (RAM), dynamic random access memory (DRAM), etc. It can also include non-volatile memory, such as storage class memory (SCM), etc., or a combination of volatile memory and non-volatile memory. The memory can be one or more. For example, the computing device may include memory corresponding to different types of processors. For example, when the computing device includes a CPU and multiple GPUs, the computing device also includes CPU memory corresponding to the CPU and GPU memory corresponding to each of the multiple GPUs.

[0079] Unlike memory, hard disks have slower read and write speeds and are typically used to store data persistently. In one embodiment, data, program instructions, etc. on the hard disk need to be loaded into memory first, and then the processor retrieves these data and / or program instructions from the memory. Hard disks include, but are not limited to, non-volatile memory (non-volatile memory), such as read-only memory (ROM), hard disk drives (HDD), or solid-state drives (SSD).

[0080] The communication interface is used to communicate with other devices, such as receiving startup instructions from other devices to start the operation of the AI ​​model.

[0081] In conjunction with the schematic diagram of the architecture of the first computing device shown in FIG4 , FIG5 shows a schematic diagram of the architecture of the second computing device:

[0082] In this second computing device, the processor specifically includes a CPU and multiple GPUs, and the memory specifically includes CPU memory corresponding to the CPU and GPU memory corresponding to each of the multiple GPUs. The CPU, the CPU memory, and any two of the multiple GPUs can be connected via a bus, such as a high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) bus. Any two GPUs can also be connected via a network, such as by communicating using a collective communication library (CCL). The GPU memory, for example, is high-bandwidth memory (HBM).

[0083] Furthermore, when the computing device runs the AI ​​model, the CPU may specifically read the AI ​​model code from the CPU memory and, based on the AI ​​model code, determine the operator to be executed and the write address corresponding to the operator. The write address is specifically the location where the operator writes generated data to the GPU memory during execution by the GPU. The CPU sends the operator and the write address corresponding to the operator to the GPU, which then executes the operator and writes the data generated during the execution of the operator to the write address corresponding to the operator.

[0084] When the CPU indicates the write address corresponding to an operator to the GPU, the CPU may specifically determine the size of the data to be stored by the GPU (denoted as data A) based on the operator. The CPU then attempts to select a memory block from the memory pool maintained by the CPU based on the size of data A. If the size of a memory block (or the size that can store data, the amount of data that can be stored, etc.) in the memory pool maintained by the CPU is greater than or equal to the size of data A, the address of physical space A (i.e., the write address corresponding to the operator) is determined based on the identifier of the memory block (denoted as memory block A) and the mapping relationship between memory block A and physical space A, and the write address corresponding to the operator is indicated to the GPU.

[0085] Furthermore, if the CPU determines that the sizes of all memory blocks in its memory pool are smaller than the size of data A, it determines physical space A' from the GPU memory based on the size of data A, maps this physical space A' to memory block A', and maintains the mapping relationship between physical space A' and memory block A'. The CPU then indicates the address of physical space A' to the GPU as the write address corresponding to the operator.

[0086] For the GPU, when executing an operator, the GPU generates data A and stores data A in memory block A (or memory block A'). The CPU can also instruct the GPU to release data A when it determines that the GPU no longer uses data A. It is understood that when the CPU determines that data A is no longer used (or dependent on) by subsequent operations, the CPU adds memory block A (or memory block A') to the memory pool.

[0087] In one possible example, the CPU may read the software code in the memory to execute the functions of the memory allocator (also known as the memory manager) and the underlying interface. In other words, the CPU may include the memory allocator and the underlying interface.

[0088] The relationship between the memory allocator, the underlying interface, the GPU, and the GPU memory can be seen in the architectural diagram of the third computing device shown in FIG6 .

[0089] The memory allocator determines the size of data A based on the operator. Furthermore, based on the size of data A, the memory allocator selects memory block A from the memory pool maintained by the memory allocator, or requests memory block A' through the underlying interface. Specifically, there are two situations:

[0090] In case (1), the memory allocator selects a memory block from the memory pool based on the size of data A (i.e., it determines that a memory block A with a size greater than or equal to the size of data A exists in the memory pool maintained by the memory allocator), and then sends the identifier of memory block A to the underlying interface. Accordingly, the underlying interface maintains the identifier of memory block A and the mapping relationship between memory block A and physical space A. Based on this mapping relationship and the identifier of memory block A, the underlying interface determines the write address corresponding to the operator and sends the write address corresponding to the operator to the GPU.

[0091] In case (2), when the memory allocator fails to select a memory block from the memory pool maintained by the memory allocator based on the size of data A (i.e., it determines that the size of all memory blocks in the memory pool maintained by the memory allocator is smaller than the size of data A), it sends a memory block request to the underlying interface, and the memory block request includes the size of data A. Correspondingly, the underlying interface determines physical space A' from the GPU memory based on the size of data A in the memory block request, maps the physical space A' to memory block A', and maintains the mapping relationship between physical space A' and memory block A'. The underlying interface returns the identifier of memory block A' to the memory allocator. Further, the memory allocator determines that memory block A' is used to store data A, and the memory allocator sends the identifier of memory block A' to the underlying interface. The underlying interface determines the write address corresponding to the operator based on the identifier of memory block A' and the mapping relationship between memory block A' and physical space A', and sends the write address corresponding to the operator to the GPU.

[0092] Furthermore, when the memory allocator determines that the data A is no longer used (or relied upon) by subsequent operations, the memory allocator adds the memory block A (or memory block A') to the memory pool.

[0093] It should be noted that the GPU in Figure 5 or Figure 6 can also be replaced with other processors for running AI models, such as other CPUs, NPUs, etc. In this application, the processors used for running AI models can be collectively referred to as AI accelerator cards. When the AI ​​accelerator card is an NPU, the CPU corresponds to the CPU memory, and the NPU corresponds to the NPU memory. The CPU can read the AI ​​model code from the CPU memory, determine the operator to be executed and the corresponding write address of the operator in the NPU memory based on the AI ​​model code, and send the operator and the corresponding write address of the operator in the NPU memory to the NPU. The NPU performs calculations based on the operator and the corresponding write address of the operator in the NPU memory. When the AI ​​accelerator card is another CPU, the two CPUs can be recorded as CPU1 and CPU2 respectively. CPU1 corresponds to CPU1 memory, and CPU2 corresponds to CPU2 memory. CPU1 can read the AI ​​model code from CPU1 memory, determine the operator to be executed and the corresponding write address of the operator in CPU2 memory based on the AI ​​model code, CPU1 sends the operator and the corresponding write address of the operator in CPU2 memory to CPU2, and CPU2 performs calculations based on the operator and the corresponding write address of the operator in CPU2 memory. Of course, the architecture applicable to this application can also have other variations, and no longer exemplify them one by one.

[0094] In the above process, if the size of the memory block used to store data is larger than the size of the data, there will be residual space in the memory block. Furthermore, if the residual space cannot be occupied by other data, it will become memory fragmentation.

[0095] For example, the CPU stores data A in memory block A, data B in memory block B, and data C in memory block C. If the size of data A is 80MB and the size of memory block A is 100MB, then the remaining space in memory block A when storing data A is 20MB. If the size of data B is 70MB and the size of memory block B is 100MB, then the remaining space in memory block B when storing data B is 30MB. If the size of data C is 50MB and the size of memory block C is 100MB, then the remaining space in memory block C when storing data C is 50MB. Furthermore, the CPU determines that the data D to be stored is 90MB. Even though the sum of the remaining space in memory blocks A, B, and C is greater than 90MB, the remaining space in these three blocks cannot be merged and cannot be used to store data D. The remaining space in memory blocks A, B, and C is memory fragmentation.

[0096] However, during the operation of AI models, the memory corresponding to AI acceleration cards (such as the above-mentioned GPU, NPU, and CPU2) is relatively tight. How to reduce memory fragmentation is a technical problem that needs to be solved urgently.

[0097] To this end, this application provides a memory management method for reducing memory fragmentation.

[0098] This memory management method can be interactively executed by a main processor and an AI accelerator card, where the main processor is, for example, a CPU. Furthermore, the methods executed by the main processor (or CPU) can be specifically divided into methods executed by a memory allocator and a method executed by an underlying interface; the AI ​​accelerator card can specifically be a GPU. For ease of description, the following uses the interaction between the CPU and GPU as an example. It can be understood that methods executed by the CPU can be replaced by methods executed by the main processor, and methods executed by the GPU can be replaced by methods executed by the AI ​​accelerator card.

[0099] It should be noted that when AI models are running, the GPU generates a lot of data, which all takes up memory blocks. In this application, this data can be divided into long-lifecycle data and short-lifecycle data. In other words, the data types of this data can include long-lifecycle data and short-lifecycle data. This data can include tensors and scalars.

[0100] In one example, data that occupies a memory block for a period greater than a time threshold is regarded as long-lifecycle type data; and data that occupies a memory block for a period less than or equal to the time threshold is regarded as short-lifecycle type data.

[0101] In another example, the following four types of data are preliminarily considered to be long-lifecycle type data, and other data except the following four types of data are preliminarily considered to be short-lifecycle type data.

[0102] FIG7 is an illustrative diagram illustrating the duration of memory block occupied by data during the operation of an AI model:

[0103] (A) Data generated during AI model initialization. For example, data such as weights and gradients generated during AI model initialization.

[0104] The data generated during the initialization of the AI ​​model may be retained until the end of the AI ​​model operation, so this data can be initially regarded as long-life cycle data.

[0105] (B) Data generated but not released during the forward operation of each iteration of the AI ​​model, such as activation values.

[0106] In other words, this data is generated during the forward operation of this iteration but is not released during the forward operation of this iteration (or is retained and used in the reverse operation of this iteration). For example, it is used for gradient calculation in the reverse operation of this iteration. Because this data is generated during the forward operation of the entire iteration and retained until the reverse operation, this data can be initially considered as long-lived data.

[0107] (C) Data generated but not released in each iteration of the AI ​​model.

[0108] That is, the data is generated in this round of iteration but is not released in this round of iteration (or is retained in the next round of iteration).

[0109] For example, some AI model codes do not perform garbage collection on some used data, causing the data to be retained until the next iteration until the data is assigned a new value.

[0110] (D) Data generated during optimizer initialization, such as optimizer status.

[0111] After the AI ​​model runs one or more rounds of iterations, the gradient converges and the optimizer begins to initialize. The data generated in this process may be retained until the end of the AI ​​model run. Therefore, this data can be initially regarded as long-lifecycle data.

[0112] Of course, the above is an example of possible long life cycle type data provided by this application. In actual operation, the user can also set other data as long life cycle type data and manage these long life cycle type data.

[0113] Furthermore, the memory allocator can manage memory pools corresponding to long life cycle types and memory pools corresponding to short life cycle types. This application refers to the memory pool corresponding to the long life cycle type as the first memory pool, and the memory pool corresponding to the short life cycle type as the second memory pool. Among them, the first memory pool records one or more memory blocks that have not been used and that the memory allocator has applied for from the memory through the underlying interface. The one or more memory blocks can be used to store data of the long life cycle type; similarly, the second memory pool records one or more memory blocks that have not been used and that the memory allocator has applied for from the memory through the underlying interface. The one or more memory blocks can be used to store data of the short life cycle type.

[0114] When the memory allocator determines that the data to be stored by the GPU is long-lived data, it may select a memory block from the first memory pool, where the size of the selected memory block is equal to the size of the data to be stored. The memory allocator then sends an identifier of the selected memory block to the underlying interface. The underlying interface determines an address of the physical space based on the identifier of the selected memory block and sends the address of the physical space to the GPU. The GPU stores the data in the physical space based on the address of the physical space (i.e., the GPU stores the data in the memory block).

[0115] When the memory allocator determines that the data to be stored by the GPU is short-lived data, it may select a memory block from the second memory pool, where the size of the selected memory block is less than or equal to the size of the data to be stored. The memory allocator then sends an identifier of the selected memory block to the underlying interface. The underlying interface determines an address of the physical space based on the identifier of the memory block and sends the address of the physical space to the GPU. The GPU stores the data in the physical space based on the address of the physical space (i.e., the GPU stores the data in the memory block).

[0116] In this application, the data to be stored of the GPU can also be referred to as data to be generated by the GPU or data to be stored determined by the memory allocator.

[0117] In this way, since data of long life cycle type occupies memory block for a long time and there is no remaining space in the memory block when the data of long life cycle type is stored in the memory block, it helps to reduce the formation of memory fragmentation during the operation of the AI ​​model.

[0118] FIG8 is a flowchart of a memory management method exemplified in this application:

[0119] Step 801: A memory allocator determines a data type of first data to be stored.

[0120] The first data is generated during the Nth iteration of the AI ​​model execution. For example, the first data is generated during the Nth iteration of the AI ​​model execution by the GPU. Alternatively, the memory allocator determines that the data to be stored during the Nth iteration of the AI ​​model execution is the first data, or the memory allocator determines that the data to be stored during the Nth iteration of the AI ​​model execution by the GPU is the first data. N is an integer greater than or equal to 2.

[0121] In one possible example, the memory allocator determines the data type of the first data to be stored. Specifically, the memory allocator determines the data type of the first data to be stored based on a generation order (or generation sequence number) of the first data in X first data and a generation order of the second data of a long-lifecycle type in X second data, where X is an integer greater than or equal to 2.

[0122] In a possible example, before determining the data type of the first data to be stored, the memory allocator may first determine the generation order of the second data of the long life cycle type in the X second data.

[0123] Specifically, the memory allocator determines X second data generated sequentially in the Mth iteration round, determines the second data of the long life cycle type from the X second data, and then determines the generation order of the second data of the long life cycle type in the X second data.

[0124] Exemplarily, the second data of the long life cycle type may be data generated but not released in the forward operation of the Mth round of iteration, or the second data of the long life cycle type may be data generated but not released in the Mth round of iteration (for details, please refer to the description in the relevant embodiment of Figure 7), where M is an integer greater than or equal to 1 and less than N.

[0125] For example, the memory allocator determines that, among the five second data items generated in the Mth iteration, the first and third second data items are long-lifecycle type second data items, and the second, fourth, and fifth second data items are short-lifecycle type second data items. Therefore, the memory allocator can determine that the long-lifecycle type second data items are generated in the order of the first and third second data items among the X second data items.

[0126] Considering that the data generated in each iteration is of the same classification, for example, in the Mth iteration, weight 1, weight 2, and activation value 1 are generated sequentially, then in the Nth iteration, weight 1, weight 2, and activation value 1 are also generated sequentially. The difference is that the values ​​of weight 1, weight 2, and activation value 1 in these two iterations are different. Accordingly, when X second data are generated sequentially in the Mth iteration, X first data are also generated sequentially in the Nth iteration, and the generation order of the long-lifecycle type second data in the X second data is the same as the generation order of the long-lifecycle type first data in the X first data.

[0127] In combination with the above example, the memory allocator may also determine that the generation order of the first data of the long life cycle type is the first and third among the X first data. Furthermore, in the above step 801, when the memory allocator determines that the first data is the first first data generated in the Nth round of iteration, it is determined that the first data is of the long life cycle type; when the memory allocator determines that the first data is the second first data generated in the Nth round of iteration, it is determined that the first data is of the short life cycle type, and so on.

[0128] In step 802 , the memory allocator determines a memory block (which may be referred to as a target memory block) for storing the first data according to the data type and size of the first data.

[0129] When the memory allocator determines the target memory block according to the data type and size of the first data, the following situations may occur:

[0130] Case 1: When the data type of the first data is a long-lifecycle type, the memory allocator selects a first memory block of the same size as the first data from the first memory pool to store the first data, that is, the target memory block is specifically the first memory block.

[0131] Case 2: When the data type of the first data is a short life cycle type, the memory allocator selects a second memory block from the second memory pool to store the first data. The size of the second memory block is greater than or equal to the size of the first data, that is, the target memory block is specifically the second memory block.

[0132] Case 3: When the data type of the first data is a long-life type and the selection of the first memory block in the first memory pool fails, the memory allocator applies for a third memory block of the same size as the first data from the memory (or physical memory, hardware memory, etc.) to store the first data, that is, the target memory block is specifically the third memory block.

[0133] Case 4: When the data type of the first data is a long-life type and the application for the third memory block from the memory fails, the fourth memory block is selected from the second memory pool to store the first data. The size of the fourth memory block is greater than or equal to the size of the first data, that is, the target memory block is specifically the fourth memory block.

[0134] In case 5, when the data type of the first data is a long-lived type and selection of the fourth memory block from the second memory pool fails, the memory allocator releases the first and second memory pools into memory. The memory allocator then requests a fifth memory block of the same size as the first data from memory to store the first data. In other words, the target memory block is specifically the fifth memory block.

[0135] In combination with the above cases 1 to 5, the following example provides a method for the memory allocator to determine the target memory block:

[0136] It can be understood that the memory allocator can select in the following priority order: (1) the memory pool corresponding to the data type of the first data, (2) the memory (specifically the memory before the memory pool is released), (3) the memory pool corresponding to other data types, and (4) the memory after the memory pool is released.

[0137] A method for determining a memory block is explained with reference to FIG9 .

[0138] In step 901, the memory allocator selects a target memory block from a memory pool corresponding to the data type of the first data according to the data type and size of the first data. Specifically, there are two possibilities:

[0139] When the memory allocator determines that the first data is of a long-life type, the memory allocator selects a target memory block from the first memory pool based on the size of the first data. It is understood that in order to avoid memory fragmentation in the first memory pool, the memory allocator needs to ensure that the size of the target memory block is equal to the size of the first data when selecting the target memory block. In addition, if the memory allocator determines that the size of all memory blocks in the first memory pool is not equal to the size of the first data, it is determined that the selection of the target memory block from the first memory pool has failed.

[0140] When the memory allocator determines that the first data is of a short life cycle type, the target memory block is selected from the second memory pool according to the size of the first data. It can be understood that since the first data is of a short life cycle type, the first data will soon be released by the GPU, that is, the first data will soon release the memory block it occupies, so the size of the target memory block can be limited to be greater than or equal to the size of the first data. Exemplarily, the target memory block is the smallest memory block in the second memory pool that is greater than the size of the first data. In addition, if the memory allocator determines that the sizes of all memory blocks in the second memory pool are smaller than the size of the first data, it is determined that the selection of the target memory block from the second memory pool has failed.

[0141] Step 902 : When the memory allocator fails to select a target memory block from the memory pool corresponding to the data type of the first data, it applies for a target memory block from the memory according to the size of the first data.

[0142] The memory allocator schedules the underlying interface to apply for a target memory block from the memory according to the size of the first data.

[0143] Specifically, the memory allocator sends a memory block request to the underlying interface based on the size of the first data, where the memory block request includes the size of the first data. Accordingly, the underlying interface determines the target physical space from the memory based on the size of the first data in the memory block request, maps the target physical space to the target memory block, and maintains the mapping relationship between the target physical space and the target memory block. Exemplarily, the size of the target memory block is equal to the size of the first data. The underlying interface returns the identifier of the target memory block to the memory allocator.

[0144] In addition, if the underlying interface determines that the free physical space in the memory cannot meet the size of the first data, a response indicating that the memory block request failed is returned to the memory allocator. Accordingly, the memory allocator determines that the target memory block request from the memory has failed.

[0145] Step 903 : When the memory allocator fails to apply for the target memory block from the memory, it selects the target memory block from the memory pool corresponding to other data types according to the size of the first data.

[0146] Here, it is only necessary to limit the size of the target memory block to be greater than or equal to the size of the first data.

[0147] There are two possible ways:

[0148] In one possible approach, if the memory allocator determines that the first data is of a long-lifecycle type, the memory allocator selects a target memory block from the second memory pool based on the size of the first data. If the memory allocator determines that the first data is of a short-lifecycle type, the memory allocator selects a target memory block from the first memory pool based on the size of the first data.

[0149] In one example of the above possible methods, considering that once the data of the long life cycle type occupies the memory block in the second memory pool, it will have a greater impact on the storage of the data of the short life cycle type, illustratively, this method can also be limited to that if the memory allocator determines that the first data is of the long life cycle type, it determines that the selection of the target memory block from the memory pool corresponding to other data types has failed.

[0150] In another example of the above possible methods, when data of short life cycle type occupies memory blocks in the first memory pool, in order to further reduce its impact on the storage of data of long life cycle type, the present application may also allow the first data to occupy memory blocks in the first memory pool only when the memory allocator determines that the first data is of short life cycle type and the first data is not generated in the forward operation of the Nth round of iteration. It is explained that since data of long life cycle type may include data generated but not released in the forward operation of each round of iteration of the AI ​​model, the data may occupy memory blocks in the first memory pool in part of a round of iteration (for example, the gradient calculation generated in the forward operation to the reverse operation), so the memory allocator may allow data of short life cycle type generated in other periods other than the forward operation (such as the reverse operation period, the optimizer initialization period, etc.) to occupy memory blocks in the first memory pool.

[0151] Step 904: If the memory allocator fails to select a target memory block from the memory pool corresponding to other data types, the memory allocator releases the first memory pool and the second memory pool into the memory. The memory allocator re-applies for the target memory block from the memory based on the size of the first data.

[0152] Here, when the memory allocator fails to select a target memory block from the memory pool corresponding to other data types, it can also release the first memory pool or the second memory pool to the memory, or release a specific memory block in the first memory pool or the second memory pool to the memory. In one possible way, the memory allocator sends a release request to the underlying interface. Exemplarily, the release request may include the identifier of the memory block to be released, and the underlying interface releases the corresponding memory block according to the identifier of the memory block to be released in the release request; another exemplary embodiment, the release request may include the identifier of the memory pool to be released, and the underlying interface releases all memory blocks in the corresponding memory pool according to the identifier of the memory pool to be released in the release request; another exemplary embodiment, the release request is used to instruct the underlying interface to release the first memory pool and the second memory pool. Unless otherwise specified, the following description takes the memory allocator (through / scheduling the underlying interface) releasing the first memory pool and the second memory pool to the memory as an example.

[0153] It should be pointed out that when the memory pool includes split memory blocks, the memory allocator will try to merge the split memory blocks first, and then schedule the underlying interface to release the merged memory blocks. For example, the memory pool allocator applies for memory block a from the underlying interface, and the size of memory block a is 100M. When the memory pool allocator determines that the size of the data a to be stored is 80M, it can split memory block a into memory block a1 and memory block a2, where the size of memory block a1 is 80M, memory block a1 is used to store data a, and memory block a2 is used as the memory block after the split. Furthermore, after memory block a1 is put back into the memory pool again, the memory allocator can merge memory block a1 and memory block a2 into memory block a, and then request the underlying interface to release memory block a.

[0154] Furthermore, as the free physical space in the memory increases, the memory allocator can re-apply for a target memory block from the memory through the underlying interface based on the size of the first data.

[0155] It should be noted that, since the memory allocator cannot determine whether the X second data are of the long-lifecycle type or the short-lifecycle type in the Mth iteration, the memory allocator can process the X second data as short-lifecycle type data. Specifically, in one possible embodiment, the memory allocator selects a sixth memory block from the second memory pool based on the size of the second data. The sixth memory block is used to store the second data, and the size of the sixth memory block is greater than or equal to the size of the second data. In another possible embodiment, if the memory allocator fails to select a sixth memory block from the second memory pool based on the size of the second data, the memory allocator sends a memory block request to the underlying interface, where the memory block request includes the size of the second data. Accordingly, the underlying interface determines physical space from memory based on the size of the second data in the memory block request, maps the physical space to a seventh memory block, and maintains the mapping relationship between the physical space and the seventh memory block. The underlying interface returns the identifier of the seventh memory block to the memory allocator. Furthermore, the memory allocator determines the seventh memory block, which is used to store the second data. For a specific implementation, please refer to the description of the method of selecting / applying for the first memory block for the first data when the memory allocator determines that the first data is of a short life cycle type.

[0156] This helps minimize memory fragmentation while ensuring the orderliness of memory management.

[0157] Optionally, it also includes: after the memory allocator determines the target memory block based on the data type of the first data and the size of the first data, it can indicate the target memory block to the GPU through the underlying interface. Exemplarily, the memory allocator sends the identifier of the target memory block to the underlying interface, and the underlying interface determines the address of the target physical space based on the identifier of the target memory block and the mapping relationship between the target memory block and the target physical space, and sends the address of the target physical space to the GPU. Accordingly, after generating the first data, the GPU can write the first data into the target physical space according to the address of the target physical space (that is, write the first data into the target memory block).

[0158] Optionally, the method further includes: Step 803, after the first data releases the target memory block, the memory allocator places the target memory block in the first memory pool or the second memory pool for management.

[0159] Exemplarily, after determining that no subsequent operation depends on the first data, the memory allocator determines that the first data releases the target memory block. That is, the GPU releases / deletes the first data, and the first data releases the target memory block occupied by it.

[0160] Exemplarily, the memory allocator places the target memory block in a memory pool for management. Specifically, the memory allocator places the target memory block as a free memory block in the memory pool for management.

[0161] When the target memory block comes from the memory pool or memory corresponding to the data type of the first data, the memory allocator places the target memory block in the memory pool corresponding to the data type of the first data for management. Exemplarily, when the data type of the first data is a long life cycle type, after the memory allocator determines that the first data releases the first memory block, it places the first memory block in the first memory pool for management (corresponding to the above situation 1); when the data type of the first data is a short life cycle type, after the memory allocator determines that the first data releases the second memory block, it places the second memory block in the second memory pool for management (corresponding to the above situation 2); when the data type of the first data is a long life cycle type, after the memory allocator determines that the first data releases the third memory block, it places the first memory block in the first memory pool for management (corresponding to the above situation 3); when the data type of the first data is a long life cycle type, after the memory allocator determines that the first data releases the fifth memory block, it places the first memory block in the first memory pool for management (corresponding to the above situation 5).

[0162] When the target memory block comes from a memory pool corresponding to another data type, the memory allocator places the target memory block in the memory pool corresponding to the other data type for management. For example, when the data type of the first data is a long-life type, if the memory allocator determines that the first data releases the fourth memory block, the memory allocator places the fourth memory block into the second memory pool for management (corresponding to the above-mentioned situation 4).

[0163] Optionally, before determining the data type of the first data, the memory allocator may also instruct the GPU to perform AI model initialization and / or instruct the GPU to perform optimizer initialization based on the AI ​​model code. The following are respectively explained:

[0164] 1. AI model initialization:

[0165] During the process of executing AI model initialization, the GPU can generate Y third data, where Y is an integer greater than or equal to 1. The memory allocator can consider all third data generated during the AI ​​model initialization process to be of a long-life type.

[0166] Among them, the third data is, for example, the weights and gradients generated when the AI ​​model is initialized.

[0167] It is understandable that before the AI ​​model is initialized, since the GPU has not yet generated the third data, the memory allocator has not yet applied for memory blocks from the memory through the underlying interface. In other words, the memory pool managed by the memory allocator is empty.

[0168] The memory allocator can apply for an eighth memory block corresponding to each third data from the memory through the underlying interface based on the long life cycle type and size of each third data. Exemplarily, the size of the eighth memory block is equal to the size of the third data. Then, when the GPU initializes the AI ​​model, it can generate the third data and, based on the address of the physical space corresponding to the eighth memory block, store the third data in the physical space corresponding to the eighth memory block (that is, store the third data in the eighth memory block).

[0169] It should be pointed out that the Y third data may actually include third data of long life cycle type and third data of short life cycle type. Although in this step, the memory allocator considers the Y third data to be of long life cycle type, and selects memory blocks from the first memory pool for the Y third data, but because the GPU has just performed AI model initialization and the memory is still relatively abundant, the memory allocator considers the third data of short life cycle type to be of long life cycle type, which has little impact on the memory. And these third data of short life cycle type will be released by the GPU in a short period of time, or it can be understood that when the memory allocator determines that the third data of short life cycle type is not relied upon by subsequent operations, it can instruct the GPU to release the third data of short life cycle type, that is, the third data of short life cycle type releases the eighth memory block occupied by it. Further, the memory allocator can re-add these eighth memory blocks to the first memory pool.

[0170] Second, optimizer initialization:

[0171] The memory allocator may determine the gradient convergence in the Kth iteration of the AI ​​model operation, where K is an integer greater than or equal to 1. Subsequently, the memory allocator instructs the GPU to perform optimizer initialization according to the AI ​​model code.

[0172] Furthermore, during the process of executing optimizer initialization on the GPU, the GPU can generate Z fourth data, where Z is an integer greater than or equal to 1. The memory allocator may consider that all fourth data generated during the optimizer initialization process are of a long-lifecycle type.

[0173] The fourth data is, for example, the optimizer state.

[0174] It is understandable that before the optimizer is initialized, the GPU has already performed AI model initialization and K rounds of iterations, and there are memory blocks in both the first memory pool and the second memory pool.

[0175] The memory allocator can select a memory block for storing the fourth data based on the long life cycle type of each fourth data and the size of the fourth data, based on the following principles: selecting from the first memory pool; if the selection from the first memory pool fails, applying from the memory based on the size of the fourth data; if the application from the memory fails, selecting from the second memory pool based on the size of the fourth data; if the selection from the second memory pool fails, releasing the first memory pool and the second memory pool to the memory, and the memory allocator again applies from the memory based on the size of the fourth data. For specific implementation, please refer to the scheme for the memory allocator to determine the target memory block in the relevant embodiment of Figure 9. Furthermore, the memory allocator can send the identifier of the selected / applied memory block to the underlying interface. The underlying interface determines the address of the physical space corresponding to the memory block based on the identifier of the memory block and the mapping relationship between the memory block and the address of the physical space, and sends the address of the physical space corresponding to the memory block to the GPU. Accordingly, when the GPU initializes the optimizer, it generates the fourth data and stores the fourth data in the physical space corresponding to the memory block based on the address of the physical space corresponding to the memory block (that is, the GPU stores the fourth data in the memory block).

[0176] In one possible example, the Kth round of iteration is after the Mth round of iteration, that is, before the memory allocator instructs the GPU to execute optimizer initialization, it can first determine the generation order of the second data of the long-lifecycle type in the X second data. The reason is that during the optimizer initialization process, a lot of long-lifecycle type data will be generated, and this data will occupy a large amount of memory, which will lead to relatively tight memory. Before the GPU executes optimizer initialization, the memory allocator will distinguish which data in each round of iteration is long-lifecycle type data and which data is short-lifecycle type data, and store them based on their respective corresponding memory pools. This helps reduce memory fragmentation, that is, it helps solve the problem of relatively tight memory during the GPU execution of optimizer initialization.

[0177] It should be noted that the Z fourth data may actually include fourth data of a long life cycle type and fourth data of a short life cycle type. These fourth data of a short life cycle type will be released by the GPU in a short period of time. Alternatively, when the memory allocator determines that the fourth data of a short life cycle type is not relied upon by subsequent operations, it can instruct the GPU to release the fourth data of a short life cycle type, that is, the fourth data of a short life cycle type releases the memory blocks occupied by them. Furthermore, the memory allocator can re-add the memory blocks occupied by the fourth data of a short life cycle type to the memory pool.

[0178] It should also be added that after the GPU executes the optimizer initialization, the data that still occupies the memory block are all long-life type data, for example, the third data of the long-life type in the AI ​​model initialization and the fourth data of the long-life type in the optimizer initialization. The memory allocator can also release the first memory pool through the underlying interface, that is, release the memory blocks in the first memory pool to the memory. For example, the memory allocator sends a release request to the underlying interface, and the release request includes the identifier of the first memory pool. Accordingly, the underlying interface releases all memory blocks in the first memory pool to the memory according to the identifier of the first memory pool in the release request.

[0179] For example, when the GPU is performing AI model initialization, the memory allocator determines that the data to be stored includes data 1 to data 3, and the sizes of data 1 to data 3 are 100MB, 80MB, and 50MB, respectively. The memory allocator can determine memory block 1 for storing data 1, memory block 2 for storing data 2, and memory block 3 for storing data 3, wherein the size of memory block 1 is 100MB, the size of memory block 2 is 80MB, and the size of memory block 3 is 50MB. When the GPU is performing optimizer initialization, the memory allocator determines that the data to be stored includes data 4 and data 5, and the sizes of data 4 and data 5 are 30MB and 20MB, respectively. The memory allocator can determine memory block 4 for storing data 4 and memory block 5 for storing data 5, wherein the size of memory block 4 is 30MB and the size of memory block 5 is 20MB. In reality, data 3 and data 5 are short-lived data. After the GPU completes optimizer initialization, data 3 and data 5 are released by the GPU. Consequently, data 3 releases memory block 3, and data 5 releases memory block 5. The memory management module then places memory blocks 3 and 5 into the first memory pool. Subsequently, the memory management module can release memory blocks 3 and 5 from the first memory pool into memory through the underlying interface.

[0180] In this way, the memory allocator can release to the memory the memory blocks in the first memory pool occupied before the third data of the short life cycle type in the above-mentioned AI model initialization, as well as the memory blocks in the first memory pool occupied before the fourth data of the short life cycle type in the above-mentioned optimizer initialization, which helps to reduce memory fragmentation and reduce memory tension.

[0181] In addition, in the above embodiment, the memory allocator may also release the second memory pool through the underlying interface, that is, release the memory blocks in the second memory pool to the memory, and the implementation method is similar to the method of releasing the first memory pool.

[0182] In particular, when the memory allocator releases the memory pool through the underlying interface, it may release the memory blocks previously occupied by some long-lifecycle data (such as data generated but not released in the forward operation of the Kth iteration, and data generated but not released in the Kth iteration). For example, the data generated but not released by the GPU in the forward operation of the Kth iteration is data 6, and the data generated but not released by the GPU in the Kth iteration is data 7. Data 6 and data 7 should be classified as long-lifecycle data. However, data 6 and data 7 may be released by the GPU after the optimizer is initialized. That is, when K is equal to M, memory block 6 and memory block 7 are placed in the second memory pool; when K is greater than M, memory block 6 and memory block 7 are placed in the first memory pool. And when the memory allocator releases the memory pool through the underlying interface, it can release memory block 6 and memory block 7.

[0183] However, it can be understood that in the K+1th iteration of the AI ​​model operation, the memory allocator can continue to re-apply for the corresponding memory blocks from the underlying interface based on the size of these long-lifecycle types of data, and when it is determined that these data are not relied upon by subsequent operations, the memory blocks corresponding to these data are placed in the first memory pool. Therefore, in the K+2th round of the AI ​​model operation, the memory blocks in the first memory pool can remain stable. Combined with the above example, in the K+1th iteration of the AI ​​model operation, the memory allocator re-applies for memory block 6 from the underlying interface based on the size of data 6, and when it is determined that data 6 is no longer relied upon by subsequent operations, memory block 6 is placed in the first memory pool; and, based on the size of data 7, the memory allocator re-applies for memory block 7 from the underlying interface, and when it is determined that data 7 is no longer relied upon by subsequent operations, memory block 7 is placed in the first memory pool. In the K+2th round of the AI ​​model operation, memory blocks 6 and 7 in the first memory pool can remain stable.

[0184] FIG10 is a flowchart of a specific memory management method provided by this application as an example:

[0185] The CPU can also read software code from memory to execute the functions of the code executor. In other words, the CPU can include a code executor. The following example uses the CPU including a code executor, memory allocator, and underlying interfaces as an example.

[0186] Step 1001: The code executor determines the second AI model code based on the marking instruction and the first AI model code.

[0187] Specifically, the code executor may receive a marking instruction from the user, and the marking instruction is used to mark the code segment associated with the long life cycle type in the first AI model code. Exemplarily, the code executor provides a marking interface to the outside, and the marking interface is used to receive the marking instruction issued by the user. Accordingly, the code executor marks the code segment associated with the long life cycle type in the first AI model code according to the marking instruction to obtain the second AI model code. In this embodiment, the second AI model code is equivalent to the AI ​​model code in Figure 8 or Figure 9.

[0188] Code segments associated with long lifecycle types include, for example, the code segment between the start and end of AI model initialization, the code segment between the start and end of forward operation, the code segment between the start and end of iteration, the code segment between the start and end of optimizer initialization, etc.

[0189] When the code executor determines the second AI model code based on the marking instruction and the first AI model code, the code executor may specifically write function decorators A to D into the first AI model code respectively to obtain the second AI model code.

[0190] The following introduces each function decorator in turn:

[0191] (1) Function decorator A is used to inform the memory allocator at the beginning and end of the AI ​​model initialization function (model initialize function). The memory allocator will consider the data generated in the AI ​​model initialization corresponding to the AI ​​model initialization function as long-life data. In addition, function decorator A is also used to traverse the initialized AI model to identify the forward function in each network layer of the AI ​​model and call function decorator B.

[0192] (2) Function decorator B is used to inform the memory allocator at the start and end of the forward function. The memory allocator will consider all data generated in the forward operation corresponding to the forward function but not released in the forward operation as long-life data. In addition, function decorator B can also be scheduled for execution other than function decorator A. For example, function decorator B is located before the start of the forward function. When the code executor executes function decorator B, it notifies the memory allocator at the start and end of the forward function.

[0193] (3) Function decorator C is used to inform the memory allocator at the start and end of the iteration function (step function). The memory allocator will consider the data generated in the iteration corresponding to the iteration function but not released in this round of iteration as long-life data.

[0194] (4) Function decorator D is used to inform the memory allocator at the beginning and end of the optimizer initialization function. The memory allocator will regard the data generated in the optimizer initialization corresponding to the optimizer initialization function as long-life data.

[0195] It should be added that the method implemented by function decorator A can also be implemented by two functions (denoted as function A1 and function A2), where function A1 is used to notify the memory allocator at the beginning of the AI ​​model initialization function; function A2 is used to notify the memory allocator at the end of the AI ​​model initialization function. Similarly, the method implemented by function decorator B (or function decorator C, or function decorator D) can also be implemented by two functions, which are used to notify the memory allocator at the beginning and end of the function respectively.

[0196] In addition, in this method, if the user wants to create a series of data that will be maintained until the end of the AI ​​model operation, the user can determine the code segment used to generate the data from the first AI model code, and add two new functions to the code segment of the first AI model code through the code executor. The two functions are used to respectively identify the beginning and end of the code segment, which helps to improve flexibility.

[0197] Step 1002: The code executor sends an AI model initialization start instruction to the memory allocator at the beginning of the AI ​​model initialization function according to the function decorator A in the second AI model code.

[0198] In step 1003, the memory allocator manages the third data to be stored (specifically, the third data generated by the GPU during the AI ​​model initialization process) according to the long lifecycle according to the AI ​​model initialization start instruction. For details, please refer to the above-mentioned Figures 8 or 9 related embodiments regarding the memory allocator's management of the third data, which will not be repeated here.

[0199] Step 1004: The code executor sends an AI model initialization end instruction to the memory allocator at the end of the AI ​​model initialization function according to the function decorator A in the second AI model code.

[0200] The purpose of steps 1002 to 1004 is that the memory allocator manages the third data to be stored in the initialization of the AI ​​model according to the long life cycle.

[0201] Step 1005: The code executor sends an iteration start instruction to the memory allocator at the beginning of the iteration function according to the function decorator C in the second AI model code.

[0202] In step 1006, the memory allocator manages the second data to be stored (specifically, the second data generated by the GPU in the first round of iteration (i.e., M=1) of executing the AI ​​model) according to a short life cycle according to the iteration start instruction.

[0203] It can be understood that step 1006 can occur at any time before step 1009.

[0204] Step 1007: The code executor sends a forward operation start instruction to the memory allocator at the start of the forward function according to the function decorator B in the second AI model code.

[0205] Step 1008: The code executor sends a forward operation end instruction to the memory allocator at the end of the forward function according to the function decorator B in the second AI model code.

[0206] In step 1009 , the code executor sends an iteration end instruction to the memory allocator at the end of the iterative function according to the function decorator C in the second AI model code.

[0207] In step 1010, the memory allocator determines the order in which the long-life second data are generated among the X second data based on the X second data generated by the GPU in the first iteration of the AI ​​model execution. X is an integer greater than or equal to 2.

[0208] The second data with a long life cycle may specifically be the second data generated but not released in the forward operation of the first round of iteration or the second data generated but not released in the first round of iteration.

[0209] In one example, the GPU generates X second data in the first round of iteration of the AI ​​model. The memory allocator can determine the generation order of the second data generated but not released in the forward operation of the first round of iteration among the X second data based on the notification of function decorator B; and determine the generation order of the second data generated but not released in the first round of iteration among the X second data based on the notification of function decorator C. For example, the GPU generates 5 second data in the first round of iteration of the AI ​​model. The memory allocator determines that the first second data is the second data generated but not released in the forward operation of the first round of iteration of the AI ​​model run by the GPU, and the third second data is the second data generated but not released in the first round of iteration of the AI ​​model run by the GPU. Therefore, the memory allocator can determine that the generation order of the second data with a long life cycle in the first round of iteration is the first and third among the 5 second data.

[0210] The purpose of steps 1005 to 1010 is that the memory allocator identifies which data in each iteration is of a long life cycle type and which data is of a short life cycle type.

[0211] In step 1011, the memory allocator determines the lifecycle type of the data to be stored in subsequent iterations according to the generation order of the long-lifecycle second data in the X second data, and manages the data according to the lifecycle type of the data to be stored.

[0212] The subsequent iteration may specifically be the second iteration or the second to Kth iterations, where K may be an integer greater than or equal to 2. Step 1011 may refer to the manner in which the memory allocator manages the first data in the above-mentioned embodiments of FIG. 8 or FIG. 9 , and will not be described in detail here.

[0213] In step 1012 , the code executor sends an optimizer initialization start instruction to the memory allocator at the start of the optimizer initialization function according to the function decorator D in the second AI model code.

[0214] In step 1013, the memory allocator manages the fourth data to be stored (specifically, the fourth data generated by the GPU during the optimizer initialization process) according to the optimizer initialization start instruction according to the long lifecycle management method. For details, please refer to the above-mentioned FIG8 or FIG9 related embodiments regarding the memory allocator's management of the fourth data, which will not be repeated here.

[0215] In step 1014, the code executor sends an optimizer initialization end instruction to the memory allocator at the end of the optimizer initialization function according to the function decorator D in the second AI model code.

[0216] The purpose of steps 1012 through 1014 is to have the memory allocator manage the fourth data to be stored during optimizer initialization according to its long lifecycle. It is understood that the code executor can execute the optimizer initialization function and function decorator D in the second AI model code after confirming gradient convergence. In other words, steps 1012 through 1014 occur after the Kth iteration.

[0217] Step 1015: The memory allocator releases the first memory pool and the second memory pool.

[0218] For specific implementation, please refer to the description of releasing the memory pool during the optimizer initialization process above.

[0219] In step 1016, the memory allocator determines the lifecycle type of the first data to be stored in the Nth iteration based on the order in which the long-lifecycle second data is generated in the X second data, and manages the first data based on the lifecycle type of the first data to be stored. For details, see the above-mentioned FIG8 or FIG9 related embodiments regarding the memory allocator's management of the first data, which will not be further described here.

[0220] Illustratively, N is greater than or equal to K+1.

[0221] In the embodiment related to FIG10 above, the AI ​​model code can run in an upper-level framework, such as DeepSpeed, Ascend Speed, Megatron, Hulk, etc.; the code executor and memory allocator can be used in a deep learning framework, such as PyTorch; the underlying interface can be located in an underlying software library, such as the Compute Architecture for Neural Networks (CANN). The relationship between the various frameworks can be seen in FIG11. The architecture of CANN can also be seen in FIG1.

[0222] The above application embodiment also needs to supplement the following three points:

[0223] First, the present application may also not divide the first memory pool and the second memory pool, but the memory allocator may uniformly manage one memory pool. When the memory allocator selects a memory block from the memory pool according to the size of the first data, if it is determined that the first data is of a long life cycle type, the size of the target memory block selected from the memory pool needs to be equal to the size of the first data; if it is determined that the first data is of a short life cycle type, the size of the target memory block selected from the memory pool is greater than or equal to the size of the first data. This method may also help reduce memory fragmentation.

[0224] In another possible example, the memory allocator may associate some memory blocks in the same memory pool with long lifecycle types, and other memory blocks with short lifecycle types. When the memory allocator selects a target memory block from the memory pool based on the size of the first data, if it is determined that the first data is of long lifecycle type, the target memory block is selected from the memory blocks associated with the long lifecycle type, and the size of the target memory block is equal to the size of the first data; if it is determined that the first data is of short lifecycle type, the target memory block is selected from the memory blocks associated with the short lifecycle type, and the size of the target memory block is greater than or equal to the size of the first data. This method can also help reduce memory fragmentation. Of course, there can be other variations in the way the memory allocator manages the memory pool, and this application will no longer give examples one by one.

[0225] Secondly, the above-mentioned method performed by the CPU and the AI ​​accelerator card can also be uniformly performed by the computing device. That is, the computing device determines the data type of the first data to be stored. The computing device determines the target memory block for storing the first data based on the data type of the first data and the size of the first data. Specifically, when the data type of the first data is a long life cycle type, the computing device selects a first memory block of the same size as the first data from the first memory pool to store the first data; when the data type of the first data is a short life cycle type, the computing device selects a second memory block from the second memory pool to store the first data, and the size of the second memory block is greater than or equal to the size of the first data, wherein the first memory pool and the second memory pool record the memory blocks requested from the memory that have not been used, and the memory blocks correspond to the physical space in the memory. After determining that the first data releases the target memory block, the computing device places the target memory block in the first memory pool or the second memory pool for management.

[0226] Third, the above description uses the operation of an AI model as an example, but this application can also be applied to other scenarios where data can be divided into long and short life cycle types, such as high performance computing (HPC) scenarios.

[0227] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the scope of protection of this application.

[0228] Based on the above content and the same concept, FIG12 is a structural diagram of a possible memory management device provided by this application.

[0229] The memory management device can be used to implement the functions of the computing device, or CPU or memory allocator in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment.

[0230] As shown in FIG12 , the memory management device 1200 includes a determination module 1201 and a memory block management module 1202 .

[0231] The determination module 1201 is configured to determine a data type of the first data to be stored.

[0232] The memory block management module 1202 is used to select a first memory block of the same size as the first data from the first memory pool to store the first data when the data type of the first data is a long life cycle type; and to select a second memory block from the second memory pool to store the first data when the data type of the first data is a short life cycle type, wherein the size of the second memory block is greater than or equal to the size of the first data; wherein the first memory pool and the second memory pool record memory blocks requested from the memory but not yet used, and the memory blocks correspond to physical space in the memory.

[0233] In one possible implementation, when the data type of the first data is a long-life type and the selection of the first memory block in the first memory pool fails, the memory block management module 1202 is also used to apply for a third memory block of the same size as the first data from the memory to store the first data; and after the first data releases the third memory block, the third memory block is placed in the first memory pool for management.

[0234] In one possible implementation, when the data type of the first data is a long-life type and the application for the third memory block from the memory fails, the memory block management module 1202 is also used to select a fourth memory block from the second memory pool to store the first data, and the size of the fourth memory block is greater than or equal to the size of the first data; after the first data releases the fourth memory block, the fourth memory block is placed in the second memory pool for management.

[0235] In one possible implementation, when the data type of the first data is a long-life type and the selection of the fourth memory block from the second memory pool fails, the memory block management module 1202 is also used to release the first memory pool and the second memory pool to the memory; and apply for a fifth memory block of the same size as the first data from the memory again to store the first data.

[0236] In a possible implementation, after the optimizer is initialized, the memory block management module 1202 is further configured to release the first memory pool and the second memory pool into the memory.

[0237] The division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0238] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for enabling a terminal device (which can be a personal computer, mobile phone, or network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0239] The descriptions of the processes corresponding to the above figures have different focuses. For parts that are not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0240] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes computer program instructions that, when loaded and executed on a computer, implement the processes or functions described in the method embodiments of the present application.

[0241] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., SSD).

[0242] In the present application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can be represented by: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of the present application, the character " / " indicates that the previous and next associated objects are in a "division" relationship.

[0243] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.

[0244] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of protection of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A memory management method, characterized in that: include: determining a data type of the first data to be stored; When the data type of the first data is a long life cycle type, selecting a first memory block having the same size as the first data from a first memory pool to store the first data; When the data type of the first data is a short-lifecycle type, selecting a second memory block from a second memory pool to store the first data, where a size of the second memory block is greater than or equal to a size of the first data; The first memory pool and the second memory pool record unused memory blocks requested from the memory, and the memory blocks correspond to physical spaces in the memory.

2. The method according to claim 1, wherein Also includes: When selection of the first memory block in the first memory pool fails, applying for a third memory block having the same size as the first data from the memory to store the first data; After the first data releases the third memory block, the third memory block is placed into the first memory pool for management.

3. The method according to claim 2, wherein Also includes: When the application for the third memory block from the memory fails, selecting a fourth memory block from the second memory pool to store the first data, where the size of the fourth memory block is greater than or equal to the size of the first data; After the first data releases the fourth memory block, the fourth memory block is placed into the second memory pool for management.

4. The method according to claim 3, wherein Also includes: When selection of the fourth memory block from the second memory pool fails, releasing the first memory pool and the second memory pool into the memory; A fifth memory block having the same size as the first data is requested from the memory to store the first data.

5. The method according to any one of claims 1 to 4, wherein The first data is data generated during the operation of the artificial intelligence (AI) model, and the memory is the memory of the AI ​​accelerator card.

6. The method according to any one of claims 1 to 5, wherein When the data type of the first data is a long life cycle type, the first data is data generated but not released in any round of iteration of the AI ​​model operation, data generated but not released in the forward operation of any round of iteration of the AI ​​model operation, data generated in the initialization of the AI ​​model, or data generated in the initialization of the optimizer.

7. The method according to claim 6, wherein Also includes: After the optimizer is initialized, the first memory pool and the second memory pool are released into the memory.

8. A memory management device, characterized in that: Comprising means for performing the method as claimed in any one of claims 1 to 7.

9. A computing device, characterized in that The computing device comprises a processor connected to a memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the computing device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instructions, and when the computer program or instructions are executed by a computing device, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a computing device, the method according to any one of claims 1 to 7 is implemented.

Images