Memory optimization method for model inference, apparatus and computing device cluster

Abstract

A memory optimization method for model inference, comprising: during the inference process of a first model, initializing a virtual memory in the first model associated with a first data item, the first data item being specified by a user; and, when the first data item is to be read / written and the virtual memory associated with the first data item has not been mapped to a physical memory, allocating the virtual memory a physical memory and mapping the virtual memory to the physical memory, and reading / writing the first data item from / to the physical memory to which the virtual memory has been mapped. In this way, during model inference processes, configuring virtual memories for specific data items can use discontinuous physical memories to cache these data items. In addition, only when these data items are required to be read / written, are the virtual memories that are associated with these data items allocated the physical memories and mapped to the physical memories, so that the allocation of the physical memories can be passively triggered in real time, thereby avoiding pre-allocating the physical memories and thus improving the utilization rate of the physical memories.

Description

A method, apparatus, and computing device cluster for optimizing model inference memory

[0001] This application claims priority to Chinese Patent Application No. 202410520380.5, filed on April 26, 2024, entitled "A Model Inference Memory Optimization Method, Apparatus and Computing Device Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus and computing device cluster for optimizing model inference memory. Background Technology

[0003] With the rapid development of large language models (LLMs), deploying LLMs for inference in real-world applications remains a key engineering challenge. LLM inference deployment involves applying a pre-trained model to a specific task, typically requiring consideration of efficiency, speed, and resource consumption. Currently, key-value cache (KV Cache) technology is commonly used in LLM inference to reuse key / value computation results and improve computational efficiency. However, the memory footprint of the KV Cache is directly proportional to the length of the generated sequence; as the generated sequence grows, the memory space required to store the KV Cache dynamically increases. Therefore, how to allocate KV Cache space during LLM inference is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a model inference memory optimization method, apparatus, computing device cluster, computer storage medium, and computer product, which can passively trigger real-time allocation of physical memory during model inference, thereby improving memory utilization.

[0005] In a first aspect, this application provides a model inference memory optimization method, comprising: during the inference process using a first model, initializing virtual memory associated with a first data item in the first model, wherein the first data item is specified by a user; when reading / writing to the first data item, and in the case where the virtual memory associated with the first data item is not mapped to physical memory, allocating and mapping physical memory to the virtual memory; and reading / writing the first data item from the physical memory mapped from the virtual memory.

[0006] In this way, during model inference, by configuring virtual memory for specific data items, these data items can be cached using non-contiguous physical memory. Simultaneously, when these data items need to be read or written, physical memory can be allocated and mapped to the virtual memory associated with them. This allows for real-time passive triggering of physical memory allocation, eliminating the need for pre-allocation and achieving on-demand allocation of physical memory, thereby improving physical memory utilization.

[0007] In one possible implementation, before inference using the first model, the method further includes: receiving configuration data from the user for the first model, wherein the configuration data specifies the first data item as a virtual Tensor, and the virtual Tensor is used to maintain the mapping relationship between virtual memory and physical memory; updating the directed acyclic graph (DAG) of the first model based on the virtual Tensor, wherein in the updated DAG, the first Tensor is replaced with a virtual Tensor, and virtual read / write operators are configured between the virtual Tensor and the operators associated with the virtual Tensor, wherein the data items associated with the first Tensor and the virtual Tensor are the same, and the virtual read / write operators are used to operate on the virtual Tensor; and obtaining the model file of the first model based on the updated DAG. In this way, by inserting virtual Tensors and virtual read / write operators into the first model, the transformation of the first model is achieved, thereby enabling the use of virtual memory to indirectly operate on physical memory during inference, and the real-time passive triggering of physical memory allocation.

[0008] In one possible implementation, updating the DAG of the first model based on a virtual Tensor includes: replacing the first Tensor in the DAG with a virtual Tensor; sequentially inserting a virtual read operator and a first temporary Tensor between the virtual Tensor and the operator that takes the virtual Tensor as input, wherein the virtual Tensor, the virtual read operator, the first temporary Tensor, and the operator that takes the virtual Tensor as input are sequentially connected, and the first temporary Tensor is used to temporarily store the output data of the operator that takes the virtual Tensor as input; and sequentially inserting a second temporary Tensor and a virtual write operator between the virtual Tensor and the operator that takes the virtual Tensor as output, wherein the operator that takes the virtual Tensor as output, the second temporary Tensor, the virtual write operator, and the virtual Tensor are sequentially connected, and the second temporary Tensor is used to temporarily store the input data of the operator that takes the virtual Tensor as output. In this way, the model transformation is achieved by adding a virtual Tensor and a virtual memory access operator to the DAG.

[0009] In one possible implementation, a model file for the first model is obtained based on the updated DAG, including: optimizing redundant virtual read / write operators in the updated DAG, and / or fusing virtual read / write operators and performance bottleneck operators in the updated DAG; the model file for the first model is then obtained based on the optimized DAG. This allows for the optimization of redundant operators and / or performance bottleneck operators in the DAG, improving the inference efficiency of subsequent models.

[0010] In one possible implementation, initializing the virtual memory associated with the first data item in the first model includes: loading the model file of the first model; initializing the physical memory block manager; and initializing the mapping table between virtual memory and physical memory maintained by the virtual Tensor in the first model. This allows for the loading of the model and the initialization of virtual and physical memory.

[0011] One possible implementation further includes: when a virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor is not mapped to physical memory, allocating physical memory for the virtual Tensor; and recording the allocated physical memory for the virtual Tensor in a mapping table between the virtual memory maintained by the virtual Tensor and physical memory. This allows for real-time triggering of physical memory allocation, enabling data to be read and written from physical memory via virtual memory.

[0012] One possible implementation further includes: when a virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor has been mapped to physical memory, executing the virtual read / write operator, and looking up the physical memory from the mapping table between the virtual memory and physical memory maintained by the virtual Tensor, and reading and writing data from the found physical memory. In this way, data can be read and written in physical memory through virtual memory.

[0013] One possible implementation also includes releasing the allocated physical memory after the first model inference ends. This avoids wasting physical memory and improves physical memory utilization.

[0014] Secondly, this application provides a model inference memory optimization apparatus, comprising: a memory initialization module and a memory allocation module. The memory initialization module is used to initialize virtual memory associated with a first data item in the first model during inference using a first model, wherein the first data item is specified by the user. The memory allocation module is used to allocate and map physical memory to the virtual memory when reading / writing to the first data item, and when the virtual memory associated with the first data item is not mapped to physical memory, and to read / write the first data item from the physical memory mapped from the virtual memory.

[0015] In one possible implementation, the module further includes a model update module. This model update module is configured to: receive configuration data from the user for the first model, wherein the configuration data specifies the first data item as a virtual tensor, and the virtual tensor is used to maintain the mapping relationship between virtual memory and physical memory; and update the directed acyclic graph (DAG) of the first model based on the virtual tensor, wherein in the updated DAG, the first tensor is replaced by the virtual tensor, and virtual read / write operators are configured between the virtual tensor and the operators associated with the virtual tensor, wherein the data items associated with the first tensor and the virtual tensor are the same, and the virtual read / write operators are used to operate on the virtual tensor; and obtain the model file of the first model based on the updated DAG.

[0016] In one possible implementation, when the model update module updates the DAG of the first model based on the virtual Tensor, it specifically performs the following steps: replacing the first Tensor in the DAG with a virtual Tensor; sequentially inserting a virtual read operator and a first temporary Tensor between the virtual Tensor and the operator that takes the virtual Tensor as input, wherein the virtual Tensor, the virtual read operator, the first temporary Tensor, and the operator that takes the virtual Tensor as input are sequentially connected, and the first temporary Tensor is used to temporarily store the output data of the operator that takes the virtual Tensor as input; and sequentially inserting a second temporary Tensor and a virtual write operator between the virtual Tensor and the operator that takes the virtual Tensor as output, wherein the operator that takes the virtual Tensor as output, the second temporary Tensor, the virtual write operator, and the virtual Tensor are sequentially connected, and the second temporary Tensor is used to temporarily store the input data of the operator that takes the virtual Tensor as output.

[0017] In one possible implementation, when the model update module obtains the model file of the first model based on the updated DAG, it is specifically used to: optimize redundant virtual read / write operators in the updated DAG, and / or, fuse virtual read / write operators and performance bottleneck operators in the updated DAG; and obtain the model file of the first model based on the optimized DAG.

[0018] In one possible implementation, when initializing the virtual memory associated with the first data item in the first model, the memory initialization module is specifically used to: load the model file of the first model; initialize the physical memory block manager; and initialize the mapping table between virtual memory and physical memory maintained by the virtual Tensor in the first model.

[0019] In one possible implementation, the memory allocation module is further configured to: allocate physical memory for the virtual Tensor when a virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor is not mapped to physical memory, and record the physical memory allocated for the virtual Tensor in a mapping table between the virtual memory maintained by the virtual Tensor and the physical memory.

[0020] In one possible implementation, the memory allocation module is further configured to: execute a virtual read / write operator when a virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor has been mapped to physical memory; and look up physical memory from the mapping table between the virtual memory maintained by the virtual Tensor and physical memory, and read and write data from the found physical memory.

[0021] In one possible implementation, the memory allocation module is also used to: release the allocated physical memory upon completion of the first model inference.

[0022] Thirdly, this application provides a model inference memory optimization apparatus, including at least one processor and an interface; at least one processor obtains program instructions through the interface; at least one processor is used to execute program line instructions to implement the method described in the first aspect or any possible implementation of the first aspect.

[0023] Fourthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0024] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0025] Sixthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0026] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0027] Figure 1 is a schematic diagram of the architecture of a model inference memory optimization system provided in an embodiment of this application;

[0028] Figure 2 is a schematic diagram of the working process of the virtual memory access operator insertion module 130 shown in Figure 1;

[0029] Figure 3 is a schematic diagram of a process for inserting a virtual Tensor and a virtual memory access operator into a DAG according to an embodiment of this application;

[0030] Figure 4 is a schematic diagram of the interaction between a tenant and a cloud computing platform provided in an embodiment of this application;

[0031] Figure 5 is a flowchart illustrating a model inference memory optimization method provided in an embodiment of this application;

[0032] Figure 6 is a schematic diagram of a model conversion process provided in an embodiment of this application;

[0033] Figure 7 is a schematic diagram of a model inference memory optimization device provided in an embodiment of this application;

[0034] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0035] Figure 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0036] Figure 10 is a schematic diagram of another computing device cluster provided in an embodiment of this application;

[0037] Figure 11 is a schematic diagram of another model inference memory optimization device provided in an embodiment of this application. Detailed Implementation

[0038] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0039] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0040] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0041] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0042] First, the technical terms used in this application will be introduced.

[0043] (1) KV Cache Technology

[0044] The working principle of KV Cache technology is to trade space for time by reusing computation results through key-value storage, thereby improving performance and reducing memory consumption. During LLM inference, the LLM calculates the corresponding key and value based on the input data and stores these results in the KV Cache. When the LLM encounters the same input data again, it can directly read the result from the KV Cache without recalculating, effectively reducing redundant computation and thus improving inference speed.

[0045] (2) Tensor

[0046] A tensor is a data structure used in AI frameworks to represent data. It is typically composed of related attributes such as shape (representing the size and dimensions of the data), data type (such as FLAOT32, int, etc.), and memory address.

[0047] (3) Operators

[0048] In the field of deep learning, an operator generally refers to a computational unit that can perform simple or complex calculations such as addition, subtraction, multiplication, and division. On acceleration hardware, these calculations are usually performed in units of operators.

[0049] The technical solution provided in this application is described below.

[0050] Generally, in LLM inference, the physical memory used for inference is required to be contiguous, and inference often involves direct manipulation of physical memory. Under this condition, two physical memory allocation methods are commonly used: the first is to pre-allocate KV Cache space according to the expected maximum length of each generated sequence, and the second is to dynamically allocate KV Cache space repeatedly. However, in the first method, there can be a significant gap between the actual physical memory used and the allocated physical memory when inferring short generated sequences, resulting in low memory utilization. In the second method, physical memory is allocated according to the actual length of the generated sequence each time inference occurs; in the next inference, the data in the previously allocated physical memory is copied to the newly allocated physical memory, and the previously allocated physical memory is released to ensure that the physical memory used for inference is contiguous. However, this method is prone to memory fragmentation when the next allocated physical memory space is larger than the released physical memory space, leading to the inability to utilize the released physical memory space.

[0051] Therefore, this application provides a model inference memory optimization method, which introduces virtual memory into the model inference process, changing the direct manipulation of physical memory during inference to indirect manipulation of physical memory through virtual memory. Since upper-layer applications directly manipulate virtual memory, and virtual memory can be infinitely large and always contiguous, mapping virtual memory to physical memory ensures normal inference operation even when physical memory is not contiguous. To achieve indirect manipulation of physical memory through virtual memory, this embodiment adds a virtual Tensor and a virtual memory access operator to the model. The virtual Tensor is used to maintain the mapping relationship between virtual memory and physical memory, and the virtual memory access operator is used to manipulate the virtual Tensor. During inference, the model only triggers the allocation and mapping of physical memory for the virtual memory maintained by the virtual Tensor when the virtual memory access operator is executed, and reads and writes data to the allocated physical memory. In this way, physical memory can be allocated and mapped in real time for the virtual memory maintained by the virtual Tensor through passive memory allocation during inference, and the allocated physical memory can be used to implement data storage, without the need to pre-allocate physical memory. This achieves on-demand allocation of physical memory, improves physical memory utilization, and decouples the developer's memory usage mode (i.e., the use of contiguous large blocks of physical memory) from the physical memory usage mode (i.e., the use of non-contiguous blocks of memory), thereby improving memory utilization and avoiding the burden on the developer.

[0052] For example, Figure 1 shows a schematic diagram of the architecture of a model inference memory optimization system provided in an embodiment of this application. As shown in Figure 1, the model inference memory optimization system 100 may include: a virtual Tensor configuration module 110, a model directed acyclic graph (DAG) construction module 120, a virtual memory access operator insertion module 130, an operator optimization module 140, a model loading module 140, a model inference module 150, a physical memory block manager 160, and physical memory 170. The virtual Tensor configuration module 110, the model DAG construction module 120, the virtual memory access operator insertion module 130, and the operator optimization module 140 are used in the model conversion stage, while the model loading module 140, the model inference module 150, the physical memory block manager 160, and the physical memory 170 are used in the model inference stage. The model conversion stage mainly involves inserting virtual memory access operators into the model, while the model inference stage mainly utilizes the model containing virtual memory access operators for inference. The various modules included in the model inference memory optimization system 100 will be described below.

[0053] The virtual tensor configuration module 110 is mainly used to receive user configurations of virtual tensors in the model and output a list of virtual tensors. Users can specify specific data items from the model (such as weights, activations (e.g., latent vectors or outputs in various network layers)) as virtual tensors based on their needs. These specified data items need to be cached using KV Cache space or other caching spaces. The virtual tensor is primarily used to maintain the mapping relationship between virtual memory and physical memory. For example, users can input configuration data into the virtual tensor configuration module 110 through a command-line window, input a configuration file containing configuration data, or configure the virtual tensors through a visual interface provided by the virtual tensor configuration module 110, etc. In this embodiment, the virtual tensor is implemented as a data storage tensor by mapping physical memory.

[0054] The model DAG construction module 120 is mainly used to construct the model's DAG based on the model description. The model description can describe the model's type (e.g., which category the model belongs to), or its specific structural details, etc. The model DAG construction module 120 can process the model description using tools or functions provided by the model-related learning framework to obtain the model's DAG. The model's DAG can include two types of nodes: operators and tensors. That is, the DAG includes data nodes and operator nodes; the edges in the DAG represent the data flow direction. For example, as shown in Figure 3(A), the nodes in this DAG include Tensors0, 1, and 2, and operators 1 and 2. The edge between Tensor0 and operator 1 points to operator 1, indicating that the data stored in Tensor0 needs to flow to operator 1.

[0055] The virtual memory access operator insertion module 130 is mainly used to insert virtual memory access operators into the DAG based on the virtual Tensor list output by the virtual Tensor configuration module 110. These virtual memory access operators are primarily used to manipulate the virtual Tensors to trigger the allocation and mapping of physical memory for the virtual memory maintained by the virtual Tensors, and to read and write data to the allocated physical memory. Virtual memory access operators can be divided into virtual write operators and virtual read operators. Virtual write operators can be used to write data to physical memory, and virtual read operators can be used to read data from physical memory. For example, as shown in Figure 2, the operation of the virtual memory access operator insertion module 130 can be as follows: In S201, the virtual Tensor list is traversed, and it is determined whether all virtual Tensors have been traversed. The process can end after all virtual Tensors have been traversed. In S202, when a virtual Tensor is encountered, it can be determined whether there exists a data item in the DAG associated with a certain Tensor that is identical to the data item associated with the virtual Tensor encountered this time. If so, S203 is executed; otherwise, execution returns to S201. In S203, the Tensors in the DAG are replaced with virtual Tensors. A virtual read operator and a temporary Tensor are sequentially inserted between the virtual Tensor and the operator that takes the virtual Tensor as input. The virtual Tensor, virtual read operator, temporary Tensor, and the operator that takes the virtual Tensor as input are sequentially connected. Simultaneously, a temporary Tensor and a virtual write operator are sequentially inserted between the virtual Tensor and the operator that takes the virtual Tensor as output. The operator that takes the virtual Tensor as output, the temporary Tensor, virtual write operator, and the virtual Tensor are sequentially connected. After executing S203, execution can return to S201. In this embodiment, the temporary Tensor is used to temporarily store data. For example, when the output of an operator is a temporary tensor, the temporary tensor is used to temporarily store the data output by the operator; when the input of an operator is a temporary tensor, the temporary tensor is used to temporarily store the data input to the operator. Regarding the operation of the virtual memory access operator insertion module 130, for example, continuing to refer to Figure 3, when the DAG is as shown in Figure 3(A), and the data item associated with Tensor1 is the same as the data item associated with the user-specified virtual Tensor1, the DAG shown in Figure 3(B) can be obtained after inserting the virtual memory access operator.

[0056] The operator optimization module 140 is mainly used to optimize the DAG of inserted virtual memory access operators to improve computation latency. For example, it optimizes redundant virtual read / write operators and merges virtual read / write operators with performance bottleneck operators. For instance, when a virtual tensor is connected to multiple virtual read operators, the virtual read operators associated with this virtual tensor can be merged; when a virtual tensor is connected to multiple virtual write operators, the virtual write operators associated with this virtual tensor can be merged. Additionally, for certain computationally intensive operators (such as the MatMul operator), they can be merged with the virtual read operators associated with these operators to optimize computation latency. Here, when the output of an operator flows to a temporary tensor, and this temporary tensor points to a virtual write operator, these three nodes can also be merged into one node. For example, continuing to refer to Figure 3, after optimizing the DAG shown in Figure 3(B), the DAG shown in Figure 3(C) can be obtained. After the operator optimization module 140 completes the optimization of the DAG containing the virtual memory access operator, the operator optimization module 140 can use tools or functions provided by the model-related learning framework to process the optimized DAG and convert it into a model file containing the virtual memory access operator. This completes the model conversion.

[0057] The model loading module 140 is primarily used to load the model file output by the operator optimization module 140 during inference, and to initialize the mapping table between virtual memory and physical memory maintained by the virtual tensors in the model. The initialized mapping table only contains virtual memory. After allocating and mapping physical memory to the virtual memory, this mapping table can record the physical memory mapped to the virtual memory. This mapping table can be stored in the virtual tensor or in the physical memory block manager 160, depending on the specific circumstances; no limitation is made here. For example, the model loading module 140 can also, but is not limited to, initialize the physical memory block manager 160.

[0058] The model inference module 150 is mainly used to perform inference on user-input questions using the model loaded by the model loading module 140. The model inference module 150 may include: an inference process judgment unit 151, an operator type judgment unit 152, an operator execution unit 153, and a physical memory allocation judgment unit 154. The inference process judgment unit 151 can be used to determine whether the inference has ended. For example, inference can be considered complete when an end marker is reached, the maximum inference length is reached, or the preset number of inference steps is reached. When the inference process judgment module 151 determines that the inference has ended, it can notify the physical memory block manager 160 to release physical memory. The operator type judgment unit 152 can be used to determine the type of operator to be executed at the current moment before the inference has ended. When the determined operator type is a non-virtual memory access operator, the operator execution unit 153 can be triggered; when the determined operator type is a virtual memory access operator, the physical memory allocation unit 154 can be triggered. The operator execution unit 153 can perform operator computations, and during execution, it can read and write data from physical memory 170. The physical memory allocation unit 154 can be used to determine whether the virtual memory maintained by the virtual tensor associated with the virtual memory access operator has been mapped to physical memory. When the virtual memory maintained by the virtual tensor has been mapped to physical memory, it indicates that physical memory has been allocated, and the operator execution unit 153 can be triggered to work; when the virtual memory maintained by the virtual tensor has not been mapped to physical memory, it indicates that no physical memory has been allocated, and the physical memory block manager 160 can be triggered to work to allocate physical memory.

[0059] The physical memory block manager 160 is primarily used to manage the physical memory 170. For example, the physical memory block manager 160 can record used and unused memory blocks, divide the physical memory 170 into blocks, and instruct the physical memory 170 to allocate or release physical memory, etc. In some embodiments, the user can pre-configure the memory block size through the physical memory block manager 160. This allows for the allocation of fixed-size blocks of physical memory each time, reducing the probability of memory fragmentation. For example, the user can set the memory block size to 4KB, resulting in subsequent allocations of (4*N)KB, where N is a positive integer. In this embodiment, the physical memory block manager 160 can instruct the physical memory 170 to release physical memory at the end of inference.

[0060] Physical memory 170 is primarily used to provide data storage space. Physical memory 170 may include a physical memory allocation unit 171 and a memory release unit 172. The physical memory allocation unit 171 can allocate physical memory based on the instructions of the physical memory block manager 160, so that virtual tensors associated with virtual memory access operators can be mapped to that physical memory. After completing the physical memory allocation, the physical memory allocation unit 171 can instruct the operator execution unit 153 to perform operator computations. Upon receiving the instructions from the physical memory allocation unit 171, the operator execution unit 153 can read and write data to the newly allocated physical memory. The memory release unit 172 can release physical memory based on the instructions of the physical memory block manager 160.

[0061] The above is an introduction to the model inference memory optimization system provided in the embodiments of this application. It is understood that the model inference memory optimization system 100 can be configured on a cloud computing platform, for example, deployed on at least one virtual machine or container instance, so that the cloud computing platform can provide model inference memory optimization services. Of course, the model inference memory optimization system 100 can also be configured on nodes other than the cloud computing platform, for example, deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud computing platform can provide pages related to public cloud services for tenants to remotely access public cloud services. In this embodiment, the tenant (also referred to as a "user") can pre-purchase the model inference memory optimization services provided by the model inference memory optimization system 100 on the cloud computing platform. For ease of understanding, the interaction between the tenant and the cloud computing platform is described below. As shown in Figure 4, the interaction between the tenant and the cloud computing platform mainly includes: the tenant logs into the cloud computing platform 400 through a client webpage, selects and purchases cloud services related to the model inference memory optimization system 100 (i.e., model inference memory optimization service) on the cloud computing platform 400. After purchase, the tenant can generate the desired algorithm on the cloud computing platform 400 based on the functions provided by the model inference memory optimization service. The cloud computing platform 400 is primarily used to manage the infrastructure for running the model inference memory optimization service. For example, the infrastructure for running the model inference memory optimization service may include multiple data centers located in different regions, each data center including multiple servers. The data centers can provide basic resources for the model inference memory optimization service, such as computing resources and storage resources. Therefore, when purchasing and using the model inference memory optimization service, the tenant mainly pays for the resources used. When using the model inference memory optimization service, the tenant can input the desired inference question through the configuration interface, application program interface (API), or other interface provided by the cloud computing platform 400. The cloud computing platform 400 can then generate an answer related to the question input by the tenant (or other software / hardware, etc.).

[0062] Alternatively, some modules in the model inference memory optimization system 100 can be configured on the cloud side and others on the edge side, thereby achieving model inference memory optimization through edge-cloud collaboration. Furthermore, the model inference memory optimization system 100 can also be entirely configured on the edge side, depending on the specific circumstances; no limitation is made here.

[0063] The above is an introduction to the model inference memory optimization system provided in the embodiments of this application. Next, based on the above content, the model inference memory optimization method provided in the embodiments of this application will be introduced.

[0064] For example, Figure 5 shows a schematic flowchart of a model inference memory optimization method provided in an embodiment of this application. It is understood that this method can be executed by any device, apparatus, platform, or device cluster with computing and processing capabilities. For example, this method can be executed by a model inference memory optimization apparatus, which can be implemented by software and / or hardware, for example, it can be configured in a computing device cluster containing at least one computing device. As shown in Figure 5, the model inference memory optimization method may include the following steps:

[0065] S501. During the inference process using the first model, the virtual memory associated with the first data item in the first model is initialized. The first data item is specified by the user.

[0066] In this embodiment, when using the first model for inference, the virtual memory associated with the first data item specified by the user in the first model can be initialized. Initializing the virtual memory can involve establishing a mapping table between virtual memory and physical memory. After initialization, since no physical memory has been allocated for the virtual memory, the mapping table may only record the virtual memory. For example, the first data item may include: the weights and / or activations of the first model, etc. The first data item requires caching.

[0067] S502, Determine whether to read / write to the first data item.

[0068] In this embodiment, during the reasoning process of the first model, the data items used in the reasoning process can be judged. If the data item used in the current reasoning is the first data item, S503 can be executed; otherwise, the reasoning continues, and the judgment continues during the reasoning process.

[0069] S503. Determine whether the virtual memory associated with the first data item is mapped to physical memory.

[0070] In this embodiment, when reading or writing the first data item, the system can query the mapping table between virtual memory and physical memory constructed during the initialization phase to see if there is any associated physical memory for the virtual memory related to the first data item. If no associated physical memory is found, it indicates that physical memory has not yet been allocated and mapped for the virtual memory related to the first data item. At this time, the allocation and mapping of physical memory for the virtual memory can be triggered, i.e., execution of S504. If an associated physical memory is found, it indicates that physical memory has been allocated and mapped for the virtual memory related to the first data item. At this time, data can be read and written from the physical memory mapped to the virtual memory, i.e., execution of S505.

[0071] S504 allocates and maps physical memory for virtual memory.

[0072] In this embodiment, when reading / writing the first data item, if it is found that the virtual memory associated with the first data item has not yet been mapped to physical memory, it indicates that physical memory has not yet been allocated to the virtual memory. Therefore, physical memory can be allocated and mapped to the virtual memory at this time, so that the first data item can be cached in physical memory, thereby indirectly operating physical memory through virtual memory.

[0073] S505, Read / write the first data item from the physical memory mapped from the virtual memory.

[0074] In this embodiment, when allocating and mapping physical memory for virtual memory associated with the first data item, the first data item can be read / written from the physical memory mapped by the virtual memory. For example, if it is the first time reading / writing the first data item, the first data item can be written to the allocated physical memory; if it is the second time reading / writing the first data item, the first data item can be read directly from the physical memory. For example, when the first model inference ends, the allocated physical memory can be released to avoid wasting physical memory and improve physical memory utilization.

[0075] In this way, during model inference, by configuring virtual memory for specific data items, these data items can be cached using non-contiguous physical memory. Simultaneously, when these data items need to be read or written, physical memory can be allocated and mapped to the virtual memory associated with them. This allows for real-time passive triggering of physical memory allocation, eliminating the need for pre-allocation and achieving on-demand allocation of physical memory, thus improving physical memory utilization.

[0076] In some embodiments, prior to S501 in Figure 5, the first model can be converted to enable indirect manipulation of physical memory using virtual memory during inference and real-time passive triggering of physical memory allocation. The conversion process for the first model, as shown in Figure 6, may include the following steps: In S601, configuration data for the first model is received from the user. This configuration data is primarily used to designate the first data item in the first model as a virtual Tensor. This virtual Tensor is mainly used to maintain the mapping relationship between virtual memory and physical memory. The user can designate certain data items in the first model as virtual Tensors according to their own needs.

[0077] In S602, the DAG of the first model is updated based on the user-specified virtual Tensor. During the update, the first Tensor in the DAG can be replaced with a virtual Tensor. The data items associated with the first Tensor and the virtual Tensor are the same. Then, a virtual read / write operator is configured between the virtual Tensor and the operator associated with the virtual Tensor. This virtual read / write operator is mainly used to operate the virtual Tensor to trigger the allocation and mapping of physical memory for the virtual memory maintained by the virtual Tensor, and to read and write data to the allocated physical memory. For example, when configuring the virtual read / write operator, a virtual read operator and a first temporary Tensor can be inserted sequentially between the virtual Tensor and the operator that takes the virtual Tensor as input, and a second temporary Tensor and a virtual write operator can be inserted sequentially between the virtual Tensor and the operator that takes the virtual Tensor as output. In this configuration, the virtual tensor, virtual read operator, first temporary tensor, and operator with virtual tensor as input are sequentially connected. Similarly, the operator with virtual tensor as output, second temporary tensor, virtual write operator, and virtual tensor are sequentially connected. The first temporary tensor is primarily used to temporarily store the output data of the operator with virtual tensor as input, and the second temporary tensor is primarily used to temporarily store the input data of the operator with virtual tensor as output. For the DAG update process of the first model, please refer to the previously described working process of the virtual memory access operator insertion module 130; it will not be elaborated upon here.

[0078] In S603, the model file of the first model is obtained based on the updated DAG. After updating the DAG of the first model, the DAG can be processed using relevant tools or functions to obtain the model file of the first model. Additionally, the updated DAG may contain some redundant operators, which may reduce the inference performance of the model. Therefore, the updated DAG can be optimized first, and then the model file of the first model can be obtained from the optimized DAG. For example, when optimizing the DAG, redundant virtual read / write operators can be merged, and / or, virtual read / write operators and performance bottleneck operators in the DAG can be fused, etc.

[0079] In this way, by inserting virtual Tensors and virtual read / write operators into the first model, the transformation of the first model is realized, and then virtual memory can be used to indirectly manipulate physical memory during inference, as well as to passively trigger the allocation of physical memory in real time.

[0080] After inserting virtual Tensors and virtual read / write operators into the first model, the model file of the first model can be loaded when S501 in Figure 5 is executed. During the loading process, the mapping table between virtual memory and physical memory maintained by the virtual Tensors in the first model is initialized, and the physical memory block manager is initialized. The physical memory block manager is mainly used for managing physical memory. For example, it records used and unused memory blocks, instructs physical memory to allocate or release physical memory, and so on.

[0081] Furthermore, during inference using the first model, when executing the virtual read / write operator, the mapping table between virtual memory and physical memory maintained by the virtual tensor can be queried to determine whether the virtual memory maintained by the virtual tensor has been mapped to physical memory. If unmapped physical memory is found, physical memory can be allocated to the virtual tensor, and this physical memory can be recorded in the mapping table maintained by the virtual tensor. If mapped physical memory is found, the virtual read / write operator can be executed, and data can be read and written from the found physical memory. In this way, data can be read and written from physical memory through virtual memory.

[0082] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0083] Based on the methods in the above embodiments, this application also provides a model inference memory optimization device.

[0084] For example, Figure 7 shows a schematic diagram of a model inference memory optimization device provided in an embodiment of this application. As shown in Figure 7, the model inference memory optimization device 700 may include a memory initialization module 710 and a memory allocation module 720. The memory initialization module 710 is used to initialize virtual memory associated with a first data item in the first model during the inference process using the first model, where the first data item is specified by the user. The memory allocation module 720 is used to allocate and map physical memory to the virtual memory when reading / writing to the first data item, and when the virtual memory associated with the first data item is not mapped to physical memory, and to read / write the first data item from the physical memory mapped from the virtual memory.

[0085] In some embodiments, the model inference memory optimization apparatus 700 further includes a model update module (not shown in the figure). The model update module is configured to: receive configuration data from a user for a first model, the configuration data specifying a first data item as a virtual tensor, the virtual tensor being used to maintain the mapping relationship between virtual memory and physical memory; and update the directed acyclic graph (DAG) of the first model based on the virtual tensor, wherein in the updated DAG, the first tensor is replaced by a virtual tensor, and a virtual read / write operator is configured between the virtual tensor and the operator associated with the virtual tensor, wherein the data items associated with the first tensor and the virtual tensor are the same, and the virtual read / write operator is used to operate on the virtual tensor; and obtain the model file of the first model based on the updated DAG.

[0086] In some embodiments, when updating the DAG of the first model based on the virtual Tensor, the model update module is specifically used to: replace the first Tensor in the DAG with the virtual Tensor; sequentially insert a virtual read operator and a first temporary Tensor between the virtual Tensor and the operator with the virtual Tensor as input, wherein the virtual Tensor, the virtual read operator, the first temporary Tensor and the operator with the virtual Tensor as input are sequentially connected, and the first temporary Tensor is used to temporarily store the output data of the operator with the virtual Tensor as input; and sequentially insert a second temporary Tensor and a virtual write operator between the virtual Tensor and the operator with the virtual Tensor as output, wherein the operator with the virtual Tensor as output, the second temporary Tensor, the virtual write operator and the virtual Tensor are sequentially connected, and the second temporary Tensor is used to temporarily store the input data of the operator with the virtual Tensor as output.

[0087] In some embodiments, when the model update module obtains the model file of the first model based on the updated DAG, it is specifically used to: optimize redundant virtual read / write operators in the updated DAG, and / or, fuse virtual read / write operators and performance bottleneck operators in the updated DAG; and obtain the model file of the first model based on the optimized DAG.

[0088] In some embodiments, when initializing the virtual memory associated with the first data item in the first model, the memory initialization module 710 is specifically used to: load the model file of the first model; initialize the physical memory block manager; and initialize the mapping table between virtual memory and physical memory maintained by the virtual Tensor in the first model.

[0089] In some embodiments, the memory allocation module 720 is further configured to: allocate physical memory for the virtual Tensor when a virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor is not mapped to physical memory, and record the physical memory allocated for the virtual Tensor in a mapping table between the virtual memory maintained by the virtual Tensor and the physical memory.

[0090] In some embodiments, the memory allocation module 720 is further configured to: execute a virtual read / write operator when a virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor has been mapped to physical memory, and look up physical memory from the mapping table between the virtual memory maintained by the virtual Tensor and physical memory, and read and write data from the found physical memory.

[0091] In some embodiments, the memory allocation module 720 is further configured to: release the allocated physical memory upon completion of the first model inference.

[0092] In some embodiments, both the memory initialization module 710 and the memory allocation module 720 shown in FIG. 7 can be implemented in software or in hardware. For example, the implementation of the memory initialization module 710 will be described below. Similarly, the implementation of the memory allocation module 720 can refer to the implementation of the memory initialization module 710.

[0093] As an example of a software functional unit, the memory initialization module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the memory initialization module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0094] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0095] As an example of a hardware functional unit, the memory initialization module 710 may include at least one computing device, such as a server. Alternatively, the memory initialization module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0096] The memory initialization module 710 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the memory initialization module 710 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the memory initialization module 710 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0097] It should be noted that, in other embodiments, the memory initialization module 710 can be used to execute any step in the model inference memory optimization method described in the above embodiments, and the memory allocation module 720 can also be used to execute any step in the model inference memory optimization method described in the above embodiments. Furthermore, the memory initialization module 710 can also be combined with the memory allocation module 720 to be responsible for executing any step in the model inference memory optimization method described in the above embodiments. In addition, the steps implemented by the memory initialization module 710 and the memory allocation module 720 can be specified as needed, and the memory initialization module 710 and the memory allocation module 720 can respectively implement different steps in the model inference memory optimization method described in the above embodiments to achieve all the functions of the model inference memory optimization device 700 shown in FIG. 7.

[0098] This application also provides a computing device 800. As shown in FIG8, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other via the bus 802. The computing device 800 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0099] Bus 802 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 804 can include pathways for transmitting information between various components of the computing device 800 (e.g., memory 806, processor 804, communication interface 808).

[0100] Processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0101] The memory 806 may include volatile memory, such as random access memory (RAM). The processor 804 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0102] The memory 806 stores executable program code, and the processor 804 executes this executable program code to implement the functions of the memory initialization module 710 and the memory allocation module 720 shown in FIG. 7, thereby realizing the model inference memory optimization method described in the above embodiments. That is, the memory 806 stores instructions for executing the model inference memory optimization method described in the above embodiments.

[0103] Alternatively, the memory 806 stores executable code, which the processor 804 executes to implement the functions of the model inference memory optimization device 700 shown in FIG. 7, thereby implementing the model inference memory optimization method described in the above embodiments. That is, the memory 806 stores instructions for executing the model inference memory optimization method described in the above embodiments.

[0104] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0105] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0106] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the model inference memory optimization method described in the above embodiments.

[0107] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the model inference memory optimization method described in the above embodiments. In other words, a combination of one or more computing devices 800 can jointly execute instructions for executing the model inference memory optimization method described in the above embodiments.

[0108] It should be noted that the memories 806 in different computing devices 800 within the computing device cluster can store different instructions, which are used to execute some of the functions of the model inference memory optimization device 700 shown in Figure 7. That is, the instructions stored in the memories 806 of different computing devices 800 can implement the functions of one or more modules in the memory initialization module 710 and the memory allocation module 720.

[0109] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 806 in computing device 800A stores instructions for performing the functions of the memory initialization module 710. Simultaneously, the memory 806 in computing device 800B stores instructions for performing the functions of the memory allocation module 720.

[0110] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0111] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 9 and 10. The difference is that the memory 806 of one or more computing devices 800 in this computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0112] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store a portion of the instructions for executing the aforementioned model inference memory optimization method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the aforementioned model inference memory optimization method.

[0113] Based on the methods in the above embodiments, this application also provides a model inference memory optimization device. Please refer to Figure 11, which is a schematic diagram of another model inference memory optimization device provided in this application. As shown in Figure 11, the model inference performance enhancement device 1100 includes one or more processors 1101 and interface circuits 1102. Optionally, the model inference performance enhancement device 1100 may also include a bus 1103. Wherein:

[0114] Processor 1101 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 1101 or through software instructions. Processor 1101 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0115] The interface circuit 1102 can be used to send or receive data, instructions or information. The processor 1101 can use the data, instructions or other information received by the interface circuit 1102 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1102.

[0116] Optionally, the model inference performance enhancement device 1100 also includes a memory, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0117] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0118] Optionally, the interface circuit 1102 can be used to output the execution results of the processor 1101.

[0119] It should be noted that the functions of the processor 1101 and the interface circuit 1102 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0120] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0121] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a cluster of computing devices including at least one computing device, the cluster of computing devices executes the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0122] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause a cluster of computing devices, including at least one computing device, to execute the methods in the above embodiments.

[0123] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0124] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0125] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0126] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for optimizing memory usage in model inference, characterized in that, The method includes: During the inference process using the first model, virtual memory associated with the first data item in the first model is initialized, and the first data item is specified by the user. When reading / writing to the first data item, and the virtual memory associated with the first data item is not mapped to physical memory, allocate and map physical memory for the virtual memory, and read / write the first data item from the physical memory mapped from the virtual memory.

2. The method according to claim 1, characterized in that, Before using the first model for inference, the method further includes: Receive configuration data from the user for the first model, wherein the configuration data is used to specify the first data item as a virtual tensor, and the virtual tensor is used to maintain the mapping relationship between virtual memory and physical memory; Based on the virtual Tensor, the directed acyclic graph (DAG) of the first model is updated, wherein in the updated DAG, the first Tensor is replaced by the virtual Tensor, and a virtual read / write operator is configured between the virtual Tensor and the operator associated with the virtual Tensor, wherein the data items associated with the first Tensor and the virtual Tensor are the same, and the virtual read / write operator is used to operate on the virtual Tensor; Based on the updated DAG, the model file of the first model is obtained.

3. The method according to claim 2, characterized in that, Based on the virtual Tensor, update the DAG of the first model, including: Replace the first Tensor in the DAG with the virtual Tensor; A virtual read operator and a first temporary Tensor are sequentially inserted between the virtual Tensor and the operator that takes the virtual Tensor as input. The virtual Tensor, the virtual read operator, the first temporary Tensor, and the operator that takes the virtual Tensor as input are sequentially connected. The first temporary Tensor is used to temporarily store the output data of the operator that takes the virtual Tensor as input. A second temporary Tensor and a virtual write operator are inserted sequentially between the virtual Tensor and the operator that outputs the virtual Tensor. The operator that outputs the virtual Tensor, the second temporary Tensor, the virtual write operator, and the virtual Tensor are connected in sequence. The second temporary Tensor is used to temporarily store the input data of the operator that outputs the virtual Tensor.

4. The method according to claim 2 or 3, characterized in that, The process of obtaining the model file of the first model based on the updated DAG includes: Optimize redundant virtual read / write operators in the updated DAG, and / or fuse virtual read / write operators and performance bottleneck operators in the updated DAG; Based on the optimized DAG, the model file of the first model is obtained.

5. The method according to any one of claims 2-4, characterized in that, The initialization of the virtual memory associated with the first data item in the first model includes: Load the model file of the first model; Initialize the physical memory block manager, and initialize the mapping table between virtual memory and physical memory maintained by the virtual Tensor in the first model.

6. The method according to claim 5, characterized in that, Also includes: When the virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor is not mapped to physical memory, physical memory is allocated to the virtual Tensor, and the physical memory allocated to the virtual Tensor is recorded in the mapping table between the virtual memory and physical memory maintained by the virtual Tensor.

7. The method according to claim 5 or 6, characterized in that, Also includes: When the virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor has been mapped to physical memory, the virtual read / write operator is executed, and the physical memory is looked up from the mapping table between the virtual memory and physical memory maintained by the virtual Tensor, and data is read and written from the found physical memory.

8. The method according to any one of claims 1-7, characterized in that, Also includes: Release the allocated physical memory once the first model inference is complete.

9. A model inference memory optimization device, characterized in that, include: The memory initialization module is used to initialize the virtual memory associated with the first data item in the first model during the inference process using the first model, wherein the first data item is specified by the user. A memory allocation module is configured to allocate and map physical memory to the virtual memory when reading / writing to the first data item and the virtual memory associated with the first data item is not mapped to physical memory, and to read / write the first data item from the physical memory mapped from the virtual memory.

10. The apparatus according to claim 9, characterized in that, Also includes: The model update module is used to receive configuration data from the user for the first model, wherein the configuration data is used to specify the first data item as a virtual tensor, and the virtual tensor is used to maintain the mapping relationship between virtual memory and physical memory; The model update module is further configured to update the directed acyclic graph (DAG) of the first model based on the virtual Tensor, wherein in the updated DAG, the first Tensor is replaced by the virtual Tensor, and a virtual read / write operator is configured between the virtual Tensor and the operator associated with the virtual Tensor, wherein the data items associated with the first Tensor and the virtual Tensor are the same, and the virtual read / write operator is used to operate on the virtual Tensor; The model update module is also used to obtain the model file of the first model based on the updated DAG.

11. The apparatus according to claim 10, characterized in that, When updating the DAG of the first model based on the virtual Tensor, the model update module is specifically used for: Replace the first Tensor in the DAG with the virtual Tensor; A virtual read operator and a first temporary Tensor are sequentially inserted between the virtual Tensor and the operator that takes the virtual Tensor as input. The virtual Tensor, the virtual read operator, the first temporary Tensor, and the operator that takes the virtual Tensor as input are sequentially connected. The first temporary Tensor is used to temporarily store the output data of the operator that takes the virtual Tensor as input. A second temporary Tensor and a virtual write operator are inserted sequentially between the virtual Tensor and the operator that outputs the virtual Tensor. The operator that outputs the virtual Tensor, the second temporary Tensor, the virtual write operator, and the virtual Tensor are connected in sequence. The second temporary Tensor is used to temporarily store the input data of the operator that outputs the virtual Tensor.

12. The apparatus according to claim 10 or 11, characterized in that, When the model update module obtains the model file of the first model based on the updated DAG, it is specifically used for: Optimize redundant virtual read / write operators in the updated DAG, and / or fuse virtual read / write operators and performance bottleneck operators in the updated DAG; Based on the optimized DAG, the model file of the first model is obtained.

13. The apparatus according to any one of claims 10-12, characterized in that, When initializing the virtual memory associated with the first data item in the first model, the memory initialization module is specifically used for: Load the model file of the first model; Initialize the physical memory block manager, and initialize the mapping table between virtual memory and physical memory maintained by the virtual Tensor in the first model.

14. The method according to claim 13, characterized in that, The memory allocation module is also used for: When the virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor is not mapped to physical memory, physical memory is allocated to the virtual Tensor, and the physical memory allocated to the virtual Tensor is recorded in the mapping table between the virtual memory and physical memory maintained by the virtual Tensor.

15. The apparatus according to claim 13 or 14, characterized in that, The memory allocation module is also used for: When the virtual read / write operator is executed and the virtual memory maintained by the virtual Tensor has been mapped to physical memory, the virtual read / write operator is executed, and the physical memory is looked up from the mapping table between the virtual memory and physical memory maintained by the virtual Tensor, and data is read and written from the found physical memory.

16. The apparatus according to any one of claims 9-15, characterized in that, The memory allocation module is also used for: Release the allocated physical memory once the first model inference is complete.

17. A model inference memory optimization device, characterized in that, Includes at least one processor and interface; The at least one processor obtains program instructions through the interface; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-8.

18. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-8.

19. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-8, wherein the cluster of computing devices includes at least one computing device.

20. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-8, wherein the computing device cluster includes at least one computing device.

Images