A method, apparatus and system for memory management

Abstract

The application discloses a memory management method applied to the field of artificial intelligence technology, and the method comprises the following steps: acquiring a calculation graph of an artificial intelligence model, the calculation graph comprising a plurality of operators and at least one edge, wherein each edge is used for representing an input relationship or an output relationship between two operators in the plurality of operators; and allocating memory for each operator in turn according to the order of the memory demand of each operator in the plurality of operators from large to small, wherein the memory address of a first operator overlaps with the memory address of a second operator, and the second operator is not the mutually exclusive operator of the first operator, and the memory of the mutually exclusive operator of the first operator cannot be reused by the first operator. The technical scheme disclosed by the application can reuse the memory of non-mutually exclusive operators, thereby improving the utilization rate of the allocated memory and reducing the peak demand of the memory occupied by the calculation graph.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a method, device and system for memory management. Background Technology

[0002] With the development of the internet, artificial intelligence (AI) has developed rapidly, and AI models are being applied to various products and services. Generally, AI models are constructed using deep neural networks or convolutional neural networks, and then trained using training samples to obtain AI models that can perform reasoning in application scenarios.

[0003] AI models can be converted into computation graphs, which include multiple operators and edges. Each edge connects two operators and represents the input / output relationship between them. During AI inference, the edge needs to compute the operators in the computation graph sequentially. Each operator has input and output data; for example, the convolution operator performs convolution on the input data and outputs the result. Therefore, each operator requires a block of memory to store its output data, which then becomes the input for the next associated operator. Without memory management for these operators, the executing operator cannot determine where to schedule input data or where the output data should be stored in memory, leading to data corruption and preventing inference. Currently, allocating a block of memory for each operator in the computation graph wastes memory resources. Summary of the Invention

[0004] This application provides a memory management method to improve memory utilization. This application also provides corresponding apparatus, computer-readable storage media, computer program products, etc.

[0005] This application provides a memory management method applied in artificial intelligence technology. The method includes: obtaining a computation graph of an artificial intelligence model, the computation graph including multiple operators and at least one edge, wherein each edge represents an input or output relationship between two operators among the multiple operators; allocating memory sequentially for each operator according to the order of memory requirements from largest to smallest, wherein the memory address of the first operator overlaps with the memory address of the second operator, and the second operator is not a mutex operator of the first operator, and the memory of the mutex operator of the first operator cannot be reused by the first operator. In other words, when allocating memory for the first or second operator, all or part of the memory from the memory of a non-mutually exclusive operator that has already been allocated memory is allocated as the memory for the first or second operator. It should be understood that "first operator" and "second operator" here are convenient terms used to refer to two non-mutually exclusive operators among the multiple operators.

[0006] Optionally, in the first aspect above, before allocating memory for multiple operators, the mutual exclusion relationship between the multiple operators can also be determined, that is, the mutual exclusion operator for each operator can be determined.

[0007] It should be noted that the order of the aforementioned steps is not limited to the order in which they are written. For example, the mutual exclusion relationship between multiple operators can be determined before memory allocation, and memory allocation can be performed using the determined mutual exclusion relationship. Alternatively, the mutual exclusion operator for the first operator can be determined during the memory allocation process, when the first operator is executed. That is, the step of determining the mutual exclusion operator can be performed before or during memory allocation. The first operator to be allocated memory, that is, the operator with the largest memory demand, may not need to determine its mutual exclusion operator, but it may be determined as the mutual exclusion operator for other operators later. The memory management method provided in this first aspect can be executed by a server that trains an artificial intelligence model, or by a terminal device that uses the artificial intelligence model. The artificial intelligence model refers to an AI model trained by the server that can be applied to the inference process. The AI ​​model can be applied to various fields such as intelligent manufacturing, intelligent transportation, smart homes, smart healthcare, smart security, autonomous driving, safe cities, and smart terminals. The device that executes this method can be a server or terminal device listed above, or it can be a virtual machine (VM) or a container.

[0008] An AI model includes multiple computational units, such as convolutional units, pooling units, and merging units, each used to perform corresponding computations. Each computational unit can be called an "operator." The execution logic of an AI model is the order in which these computational units are executed. The computational units in an AI model and the input / output relationships between them can be represented by a graph called a "computation graph." This computation graph includes all the operators in the AI ​​model. In the computation graph, the input / output relationship between two operators is represented by "edges." In the computation graph, two operators connected by an edge have a dependency relationship. This dependency means that one operator outputs data to another operator connected by the edge, and the other operator receives data input from the first operator. Therefore, this dependency relationship is also described as an input / output relationship between the two operators.

[0009] Because operators in the computation graph are executed sequentially, when a later operator is executed, the memory of an earlier operator may have already been cleared. Thus, a later operator can use all or part of the memory of an earlier operator. However, some memory of an earlier operator cannot be used by a later operator; these earlier operators whose memory cannot be used by later operators are called mutexes. A mutex can also be understood as an operator whose memory usage is mutually exclusive with the first operator; the memory of a mutex cannot be used by the first operator.

[0010] In this application, when allocating memory for multiple operators in a computation graph, the principle of prioritizing large memory is followed, that is, memory is allocated first to the operator with the largest memory requirement, and then memory is allocated to each operator in the computation graph in descending order of memory requirement.

[0011] In this application, the memory addresses of the first operator and the second operator overlap, including partial or complete overlap. If the memory requirement of the first operator is less than that of the second operator, then the memory of the second operator is allocated before that of the first operator. Thus, when allocating memory for the first operator, if the second operator is not a mutex operator of the first operator, then the first operator can use the memory of the second operator, and the memory of the second operator can be allocated to the first operator. Because the first operator allocates memory after the second operator, the memory requirement of the first operator is less than or equal to that of the second operator. If the memory requirement of the first operator is the same as that of the second operator, then all the memory of the second operator is allocated to the first operator, thus the memory addresses of the first and second operators completely overlap. If the memory requirement of the second operator is less than that of the first operator, then a portion of the memory is allocated from the memory of the second operator to the first operator, thus the memory addresses of the first and second operators partially overlap. Conversely, if the memory requirement of the first operator is greater than that of the second operator, then the memory for the first operator is allocated before that for the second operator. Thus, the execution process for allocating memory for the second operator can be understood by referring to the execution process for allocating memory for the first operator, except that the second operator is replaced by the first operator, and vice versa.

[0012] As can be seen from the first aspect of the solution above, when allocating memory for an operator in the computation graph, memory can be allocated from operators that are not mutually exclusive with the operator and have already been allocated memory. In this way, it is not necessary to apply for memory separately for the operator, which can improve the utilization rate of the allocated memory and reduce the peak memory requirement of the computation graph.

[0013] In one possible implementation of the first aspect, the mutex operator includes an operator that has an input relationship or an output relationship with the first operator.

[0014] In this possible implementation, because the two operators connected by the same edge have different memory structures—the output operator needs to store the input and output data in memory, while the receiving operator needs to read data from the output operator's memory and store it in its own—their memory cannot be reused. Therefore, these two operators with an input or output relationship become mutually exclusive operators. This possible implementation provides the idea of ​​mutual exclusion operators, thus avoiding interference between operators when reusing memory.

[0015] In one possible implementation, the mutex operator further includes at least one of the following two types of operators: an operator on a second branch in the computation graph that is parallel to the first branch where the first operator is located, and a network output operator that is located before the first operator in the execution order, wherein the network output operator is an operator that has an input relationship with other operators in the computation graph but no output relationship.

[0016] In this possible implementation, the computation graph often contains multiple parallel branches, which refer to branches originating from a single operator. These branches may also converge at another operator. Of course, these branches may not converge at another operator. In this application, whether or not these branches converge does not affect the term "parallel branches." The network output operator refers to the output operator of the entire computation graph. Because the operators in parallel branches execute their computations in parallel and all use memory, the memory of operators on parallel branches cannot be reused. Furthermore, the network output operator needs to continuously store its output results; therefore, the memory of the network output operator cannot be reused by other operators. Thus, this possible implementation considers the case of multiple mutually exclusive operators, which is beneficial for improving the accuracy of the inference process.

[0017] In one possible implementation, the above steps are as follows: memory is allocated to each operator in descending order of memory requirement, including: if the first operator is the operator with the largest memory requirement among the multiple operators, then memory is requested for the first operator according to its memory requirement.

[0018] In this possible implementation, the operator with the largest memory requirement is the first operator to be allocated memory. Therefore, there is no possibility of reusing memory from other operators. So, memory can be directly allocated for the operator with the largest memory requirement.

[0019] In one possible implementation, the above steps are as follows: memory is allocated to each operator in descending order of memory requirement, including: if the first operator is not the operator with the largest memory requirement among the multiple operators, then the memory address of each operator whose memory has been allocated is obtained; memory is allocated to the first operator from the memory of the non-mutually exclusive operators whose memory has been allocated, where the non-mutually exclusive operators are the operators other than the mutexes of the first operator among the multiple operators, and the non-mutually exclusive operators include the second operator.

[0020] In this possible implementation, if the first operator is not the one with the largest memory requirement, it means that the first operator is not the first operator to allocate memory. In this case, since some memory has already been allocated for the computation graph, we can first consider whether there is any reusable memory for the first operator among the allocated memory. If there is memory for a non-mutually exclusive operator of the first operator among the allocated memory, then memory can be allocated for the first operator from the memory of that non-mutually exclusive operator. In this way, we achieve the reuse of allocated memory without affecting the data stored in memory, thus improving memory utilization.

[0021] In one possible implementation, the above step of allocating memory for the first operator from the memory of the non-mutually exclusive operator with allocated memory includes: if the remaining memory of the non-mutually exclusive operator with allocated memory meets the memory requirement of the first operator, then the memory corresponding to the memory requirement is allocated to the first operator from the remaining memory of the non-mutually exclusive operator. The remaining memory of the non-mutually exclusive operator is either the total memory allocated to the non-mutually exclusive operator or the memory remaining after being occupied by the third operator from the total memory of the non-mutually exclusive operator. The third operator is an operator that has been allocated memory before the first operator and is not mutually exclusive with the non-mutually exclusive operator.

[0022] In this possible implementation, because the memory requirements of operators allocated earlier are larger and those allocated later are smaller, it's possible that the memory of an operator allocated earlier might be reused by multiple operators allocated later. If the remaining memory of the previously allocated non-mutually exclusive operator can meet the memory requirements of the first operator, then it can be allocated directly. In this application, the memory of operators allocated later can be allocated from the memory of one previously allocated non-mutually exclusive operator, or from the memory of multiple previously allocated non-mutually exclusive operators. The memory of these later-allocated operators can be scattered across the memory of multiple non-mutually exclusive operators. This allows for the reuse of allocated memory as much as possible, further improving memory utilization.

[0023] In one possible implementation, the method further includes: if the remaining memory of the non-mutually exclusive operator whose allocated memory does not meet the memory requirements of the first operator, then the remaining memory is allocated to the first operator, and additional spare memory is requested for the first operator, the value of which is the difference between the memory requirements of the first operator and the remaining memory of the non-mutually exclusive operator.

[0024] In this possible implementation, if the remaining memory of the previously allocated non-mutually exclusive operator cannot meet the memory requirements of the first operator, the remaining memory can be allocated first, and then the insufficient memory can be requested. This way, the allocated memory can be reused as much as possible, further improving memory utilization.

[0025] In one possible implementation, the method further includes sending the memory address of each of a plurality of operators to a terminal device, the memory address of each operator indicating the memory used by each operator when the terminal device performs data inference using an artificial intelligence model.

[0026] In this possible implementation, if the memory management process is performed by a server used for model training, the memory address corresponding to each operator can be sent to the terminal device. This allows the terminal device to use the appropriate memory based on the memory address of each operator during inference. This approach saves the resources consumed by the terminal device for memory management of each operator.

[0027] A second aspect of this application provides a memory management apparatus that has the functionality to implement the method described in the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described functionality, such as an acquisition unit and an allocation unit.

[0028] A third aspect of this application provides a computer device including at least one processor, a memory, an input / output (I / O) interface, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0029] The fourth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0030] The fifth aspect of this application provides a computer program product that stores one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0031] A sixth aspect of this application provides a chip system including at least one processor. The at least one processor is used by means of supporting memory management to implement the functions involved in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing program instructions and data necessary for the memory management means. This chip system may be composed of chips or may include chips and other discrete devices.

[0032] In this embodiment of the application, when allocating memory for an operator in a computation graph, memory can be allocated for the operator from operators that are not mutually exclusive with the operator and have already been allocated memory. In this way, it is not necessary to apply for memory separately for the operator, which can improve the utilization rate of the allocated memory and reduce the peak memory requirement occupied by the computation graph. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of another application scenario provided by an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of an embodiment of the memory management method provided in this application;

[0036] Figure 4 This is an example schematic diagram of a computation graph provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of a memory allocation scenario provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram of an embodiment of the memory management apparatus provided in this application;

[0039] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0040] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will understand, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0041] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] This application provides a memory management method to improve memory utilization. This application also provides corresponding apparatus, computer-readable storage media, computer program products, etc., which will be described in detail below.

[0043] The memory management method provided in this application embodiment is applicable to artificial intelligence (AI) technology.

[0044] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0046] The memory management method provided in this application can be applied to intelligent manufacturing, intelligent transportation, smart home, smart healthcare, intelligent security, autonomous driving, safe city, or smart terminals, etc.

[0047] The application of artificial intelligence in various scenarios is typically achieved through AI models performing reasoning on data. AI models are usually trained on the model owner's device or platform (such as a server, virtual machine (VM), or container), and the trained model is stored as a model file. When a model user's device (such as a terminal device, server, edge device, VM, or container) needs to use the AI ​​model, the user's device can either actively load the model file, or the model owner's device can proactively send the model file to the user's device for installation, and then the user's device uses the AI ​​model to perform reasoning on the data.

[0048] A server refers to a physical machine.

[0049] Terminal equipment (also known as user equipment, UE) is a device with wireless transceiver capabilities. It can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it can also be deployed on water (such as on ships); and it can be deployed in the air (such as on airplanes, balloons, and satellites). The terminal can be a mobile phone, tablet, computer with wireless transceiver capabilities, virtual reality (VR) terminal, augmented reality (AR) terminal, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical care, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc.

[0050] Both VMs and containers are virtualized devices created by virtualizing the hardware resources of a physical machine.

[0051] An AI model includes multiple computational units, such as convolutional units, pooling units, and merging units, each type of unit performing corresponding computations. Each computational unit can be called an "operator." The execution logic of an AI model is the order in which these computational units are executed. The computational units in an AI model and the input / output relationships between them can be represented by a graph called a "computation graph." This computation graph includes all the operators in the AI ​​model. In the computation graph, the input / output relationship between two operators is represented by "edges." In the computation graph, two operators connected by an edge have a dependency relationship. This dependency relationship means that one operator outputs data to another operator on that edge, and the other operator receives data input from the first operator. Therefore, this dependency relationship is also described as an input / output relationship between the two operators.

[0052] Because each operator in the computation graph produces output data during inference, each operator requires a block of memory to store its output data. Allocating a separate block of memory for each operator, especially since this memory isn't always in use, would undoubtedly waste memory resources. Therefore, this application provides a memory management method to reduce memory resource waste. The memory management method provided by this application is described below with reference to the accompanying drawings.

[0053] The memory management method provided in this application can be executed on a server or on a terminal device. If executed on a server, the process can be found in [reference needed]. Figure 1 To understand this, please refer to the following: If executed on a terminal device, the procedure can be found here. Figure 2 To understand.

[0054] like Figure 1 As shown, the server trains the initial AI model to obtain the target AI model. Then, the server converts the target AI model into a computation graph, allocates the required memory for each operator in the computation graph to the terminal device, and records the memory address of each operator. When sending the target AI model, the server also sends the memory address of each operator in the computation graph to the corresponding terminal device. In this way, when the terminal device uses the target AI model for inference, it can use the memory addresses of each operator to access the corresponding memory. The range of the memory addresses of the operators in this application can be represented using offset addresses and address offset lengths.

[0055] like Figure 2 As shown, the server trains the initial AI model to obtain the target AI model. Then, the server sends this target AI model to the terminal device. Upon receiving the target AI model, the terminal device converts it into a computation graph, allocates the required memory for each operator in the computation graph, and records the memory address of each operator. In this way, when the terminal device uses the target AI model for inference, it can use the memory addresses of each operator to access the corresponding memory.

[0056] The above Figure 1 or Figure 2 In the provided scenarios, whether the server or the terminal device manages the operator's memory, the memory management process is essentially the same. The difference lies in the server's role: if the server manages the memory, it needs to obtain the address of the available memory space in the corresponding terminal device and then send the memory address of each operator to the terminal device. The following describes the management process that is executed by both the server and the terminal device. This process can be found in [reference needed]. Figure 3 To understand.

[0057] like Figure 3 As shown, one embodiment of the memory management method provided in this application includes:

[0058] 101. Obtain the computation graph of the artificial intelligence model, which includes multiple operators and at least one edge.

[0059] Each edge represents the input or output relationship between two operators in a set of multiple operators.

[0060] Artificial intelligence models can be converted into the form of computational graphs. The computational units in the artificial intelligence model are operators in the computational graph, and the relationships between the computational units in the artificial intelligence model are edges in the computational graph.

[0061] The calculation diagram can be found here. Figure 4 To understand. For example Figure 4 The computational graph shown includes eight operators, numbered 1 through 8. Figure 4 There are multiple edges, each connecting two operators and representing the input or output relationship between them. For example, if there is an edge between operator 1 and operator 2, it means that operator 1 and operator 2 have an input or output relationship. Figure 4 If the execution order starts from operator 1, it means that the output data of operator 1 will be input to operator 2.

[0062] 102. Allocate memory to each operator in descending order of their memory requirements.

[0063] In this case, the memory address of the first operator overlaps with the memory address of the second operator, and the second operator is not a mutex operator of the first operator. The memory of the mutex operator of the first operator cannot be reused by the first operator.

[0064] Because operators in the computation graph are executed sequentially, when a later operator is executed, the memory of an earlier operator might have already been cleared. Thus, a later operator can use the memory of an earlier operator. However, some memory of an earlier operator cannot be used by a later operator; these earlier operators are called mutexes. A mutex can also be understood as an operator whose memory usage is mutually exclusive with the first operator; the memory of a mutex cannot be used by the first operator.

[0065] In this embodiment of the application, when allocating memory for multiple operators in the computation graph, the principle of prioritizing large memory is followed, that is, memory is allocated first to the operator with the largest memory requirement, and then memory is allocated to each operator in the computation graph in descending order of memory requirement.

[0066] In this embodiment, the memory addresses of the first operator and the second operator overlap, including partial or complete overlap. If the memory requirement of the first operator is less than that of the second operator, the memory of the second operator is allocated before that of the first operator. Thus, when allocating memory for the first operator, if the second operator is not a mutex to the first operator, the first operator can use the memory of the second operator, and the memory of the second operator can be allocated to the first operator. Because the first operator allocates memory after the second operator, the memory requirement of the first operator is less than or equal to that of the second operator. If the memory requirement of the first operator is the same as that of the second operator, all the memory of the second operator is allocated to the first operator, resulting in complete overlap between the memory addresses of the first and second operators. If the memory requirement of the second operator is less than that of the first operator, a portion of the memory is allocated from the memory of the second operator to the first operator, resulting in partial overlap between the memory addresses of the first and second operators. Conversely, if the memory requirement of the first operator is greater than that of the second operator, then the memory for the first operator is allocated before that for the second operator. Thus, the execution process for allocating memory for the second operator can be understood by referring to the execution process for allocating memory for the first operator, except that the second operator is replaced by the first operator, and vice versa.

[0067] In summary, when allocating memory for the first or second operator, all or part of the memory from the memory of the non-mutually exclusive operator that has already been allocated memory is used as the memory for the first or second operator. It should be understood that "first operator" and "second operator" here are convenient terms used to refer to two non-mutually exclusive operators among multiple operators.

[0068] It should be noted that the order of the aforementioned steps is not limited to the order stated in the text. For example, the mutual exclusion relationship between multiple operators can be determined before memory allocation, and the determined mutual exclusion relationship can be used during memory allocation. Alternatively, the mutual exclusion operator for the first operator can be determined during the memory allocation process, when the first operator is executed. That is, the step of determining the mutual exclusion operator can be performed before or during memory allocation. The first operator to be allocated memory, that is, the operator with the largest memory requirement, may not need to have its mutual exclusion operator determined, but it may be determined as the mutual exclusion operator for other operators later.

[0069] In this embodiment of the application, when allocating memory for an operator in a computation graph, memory can be allocated for the operator from operators that are not mutually exclusive with the operator and have already been allocated memory. In this way, it is not necessary to apply for memory separately for the operator, which can improve the utilization rate of the allocated memory and reduce the peak memory requirement occupied by the computation graph.

[0070] In step 102 above, a mutex operator is determined for the first operator. This mutex operator can include three types, which will be described below.

[0071] 1. Mutual exclusion operators include operators that have an input relationship or an output relationship with the first operator.

[0072] In this case, two operators connected by the same edge are mutually exclusive operators. For example... Figure 4 In the expression, operator 1 and operator 2 are mutually exclusive operators, operator 2 and operator 3 are mutually exclusive operators, operator 2 and operator 4 are mutually exclusive operators, operator 2 and operator 5 are mutually exclusive operators, operator 3 and operator 6 are mutually exclusive operators, operator 4 and operator 7 are mutually exclusive operators, operator 6 and operator 7 are mutually exclusive operators, and operator 6 and operator 8 are mutually exclusive operators.

[0073] 2. Calculate the operators on the second branch that are parallel to the first branch containing the first operator in the graph.

[0074] In computational graphs, there are often multiple branches running in parallel. Parallel branches refer to multiple branches originating from a single operator. These branches may also converge at another operator, such as... Figure 4 Between operators 2 and 7, there are two parallel branches: one branch contains operators 3 and 6, and the other branch contains operator 4. Of course, these branches may not converge in the other operator; whether they converge or not does not affect the fact that multiple branches are called parallel branches. For example... Figure 4 In the example, node 5 can also be a parallel branch of operator 3 and operator 6, or operator 4.

[0075] Thus, in Figure 4 In the example shown, operator 3 and operator 4 are mutually exclusive operators, operator 6 and operator 4 are mutually exclusive operators, operator 5 and operator 3 are mutually exclusive operators, operator 5 and operator 4 are mutually exclusive operators, and operator 5 and operator 6 are mutually exclusive operators.

[0076] 3. Network output operators that are executed before the first operator in the order of execution. Network output operators are operators that have input relationships with other operators in the computation graph but no output relationships.

[0077] In this embodiment of the application, the network output operator refers to the output operator of the entire computation graph. For example... Figure 4 Operators 5 and 8 in the code are both network output operators.

[0078] Figure 4The computation graph shown starts execution from operator 1. When it reaches operator 2, operator 2 has three outputs, which are sent to operators 3, 4, and 5 respectively. Operator 3 also passes its output data to operator 6, and operator 4 passes its output data to operator 7. Operator 5 is not connected to any other operators and does not need to output data to other operators. Therefore, the memory corresponding to operator 5 needs to store its output data. To prevent the output data of operator 5 from being overwritten by other operators, operators executed after operator 5 cannot use the memory corresponding to operator 5. Figure 4 In the context of the code, the operators executed after operator 5 are operators 6, 7, and 8. Therefore, operators 6, 7, and 8 cannot use the memory of operator 5. Thus, the mutex operators of operators 6, 7, and 8 all include operator 5.

[0079] exist Figure 4 In the scenario shown, if the above three types of conditions are met, based on the above analysis, Figure 4 The mutual exclusion operators of the eight operators can be understood by referring to Table 1.

[0080] Table 1: Mutual Exclusion Operators of the First Operator

[0081] First Operator Mutual exclusion operator Operator 1 Operator 2 Operator 2 Operator 1, Operator 3, Operator 4, Operator 5 Operator 3 Operator 2, Operator 4, Operator 5, Operator 6 Operator 4 Operator 2, Operator 3, Operator 5, Operator 6, Operator 7 Operator 5 Operator 2, Operator 3, Operator 4, Operator 6, Operator 7, Operator 8 Operator 6 Operator 3, Operator 4, Operator 5, Operator 7 Operator 7 Operator 4, Operator 5, Operator 6, Operator 8 Operator 8 Operator 7, Operator 5

[0082] After determining the mutual exclusion operators for each operator in the computation graph, the operators can be sorted according to their memory requirements. Figure 4 The memory requirements for the eight operators can be understood by referring to the example in Table 2.

[0083] Table 2: Memory Requirements of Operators

[0084] Operator name Memory requirements (megabytes: MB) Operator 1 5 Operator 2 10 Operator 3 8 Operator 4 20 Operator 5 2 Operator 6 6 Operator 7 15 Operator 8 3

[0085] As shown in Table 2, the memory requirements of the eight operators are ordered from largest to smallest as follows: Operator 4 (20M) > Operator 7 (15M) > Operator 2 (10M) > Operator 3 (8M) > Operator 6 (6M) > Operator 1 (5M) > Operator 8 (3M) > Operator 5 (2M).

[0086] Next, memory can be allocated sequentially according to the order of the eight operators. If the first operator to be allocated memory is the operator with the largest memory requirement in the computation graph, such as operator 4 above, then this means that the first operator is the first operator to allocate memory in the computation graph. Since the memory pool corresponding to this computation graph currently has 0 memory, memory can be directly allocated to this first operator. For example... Figure 5 In this context, 20MB of memory is allocated for operator 4. After memory allocation for operator 4, its memory address is recorded.

[0087] If the first operator to be allocated memory is not the operator with the largest memory requirement in the computation graph, then obtain the memory address of each operator that has already allocated memory; allocate memory for the first operator from the memory of the non-mutually exclusive operators that have already allocated memory. The non-mutually exclusive operators are the operators other than the mutex operators of the first operator among the multiple operators, including the second operator.

[0088] In this embodiment of the application, if the remaining memory of the non-mutually exclusive operator that has been allocated memory meets the memory requirements of the first operator, then the memory required by the first operator is allocated from the remaining memory of the non-mutually exclusive operator. The remaining memory of the non-mutually exclusive operator is either the total memory allocated to the non-mutually exclusive operator or the memory remaining after the third operator occupies the total memory of the non-mutually exclusive operator. The third operator is an operator that has been allocated memory before the first operator and is not mutually exclusive with the non-mutually exclusive operator.

[0089] In this embodiment of the application, if the remaining memory of the non-mutually exclusive operator that has been allocated memory does not meet the memory requirements of the first operator, the remaining memory is allocated to the first operator, and additional spare memory is requested for the first operator. The value of the spare memory is the difference between the memory requirements of the first operator and the remaining memory of the non-mutually exclusive operator.

[0090] In combination with the above Figure 4 Of the eight operators, after operator 4 has allocated 20MB of memory, memory will continue to be allocated for operator 7, which is ranked second. For example... Figure 5 As shown, at this point, the memory pool already has 20MB of memory allocated for operator 4. Referring to Table 1, it can be determined that operator 7 and operator 4 are mutually exclusive operators. Therefore, operator 7 cannot use the 20MB of memory allocated to operator 4, and an additional 15MB of memory needs to be allocated for operator 7. After allocating memory for operator 7, its memory address is recorded.

[0091] Next, memory will be allocated for operator 2, which is the third sorted operator. For example... Figure 5 As shown, at this point, the memory pool already contains 35MB of memory allocated for operators 4 and 7. Referring to Table 1, it can be determined that the mutex operator of operator 2 includes operator 4 but excludes operator 7. Therefore, the required 10MB of memory can be allocated from operator 7 for operator 2. At this point, it is only necessary to obtain the memory addresses of operators 4 and 7, then exclude the memory address of the mutex operator 4 of operator 2, allocate the corresponding memory for operator 2 based on the memory address of operator 7, and record the memory address of operator 2.

[0092] Next, memory will be allocated for operator 3, which is ranked fourth. For example... Figure 5As shown, at this point, the memory pool already has 35MB of memory allocated for operators 4 and 7, and operator 2 reuses 10MB of memory from operator 7. Referring to Table 1, it can be determined that the mutex operators for operator 3 include operators 4 and 2, but not operator 7. Currently, 10MB of memory in operator 7 has been reused by operator 2, leaving only 5MB of free memory. Operator 3 requires 8MB of memory, leaving a 3MB shortfall. Therefore, another 3MB of memory can be allocated for operator 3, and the addresses of the 5MB from operator 7 and the newly allocated 3MB of memory for operator 3 should be recorded. This will bring the memory pool to 38MB.

[0093] Next, memory will be allocated for operator 6, which is the fifth operator in the sorting order. For example... Figure 5 As shown, at this point, the memory pool already contains 38MB of memory. Referring to Table 1, it can be determined that the mutex operators of operator 6 include operators 3, 4, and 7, which have already allocated memory, but not operator 2, which has already allocated memory. However, operator 2 reuses the memory of operator 7. Therefore, the current 38MB of memory in the memory pool does not include the memory for the non-mutexistent operators of operator 6. An additional 6MB of memory needs to be allocated for operator 6, and the memory address of operator 6 should be recorded. After allocating this 6MB of memory, the memory pool will contain 44MB of memory.

[0094] Next, memory will be allocated for operator 1, which is sixth in the sorting order. For example... Figure 5 As shown, at this point, the memory pool already contains 44MB of memory. Referring to Table 1, it can be determined that operator 1's mutex operator is only operator 2. Memory can be allocated for operator 1 from the memory of other non-mutually exclusive operators already allocated, such as allocating 5MB of memory from operator 1 and recording the memory address of operator 1. Because the memory was allocated using the memory reuse method of operator 4, the memory pool still contains 44MB of memory.

[0095] Next, memory will be allocated for operator 8, which is the seventh operator in the sorting order. For example... Figure 5 As shown, at this point, there is already 44M of memory in the memory pool. Based on Table 1, it can be determined that operator 8 has mutex operators 7 and 5. Currently, only operator 7 has allocated memory in the memory pool. Therefore, 3M of memory can be allocated for operator 8 from other non-mutex operators, such as allocating 3M of memory for operator 8 from the memory of operator 4. Figure 5 The diagram shows a 3MB memory allocation for operator 8, excluding the memory used by operator 1 within operator 4. Since operator 1 and operator 8 are not mutually exclusive, 3MB of memory can also be allocated for operator 8 within the 5MB memory reused by operator 1. After allocating memory for operator 8, its memory address is recorded. Because the memory is allocated using the memory reuse method of operator 4, the memory pool still contains 44MB of memory.

[0096] Next, memory will be allocated for operator 5, which is ranked eighth. For example... Figure 5As shown, at this point, the memory pool already contains 44MB of memory. Referring to Table 1, we can determine that the mutex operators for operator 5 are operators 2, 3, 4, 6, 7, and 8. Only operator 1 is not a mutex operator for operator 5, but operator 1 reuses the memory of operator 4. Therefore, we need to allocate another 2MB of memory for operator 5. After allocating 2MB of memory, we record the memory address of operator 5. At this point, the memory pool contains 46MB of memory.

[0097] From the above Figures 4 to 5 As can be seen from the scenario, the total memory requirement of the 8 operators in the computation graph is 20+15+10+8+6+5+3+2=69M. However, by using the memory reuse method of non-mutually exclusive operators in this application, only 46M of memory is needed to complete the memory allocation of each operator in the computation graph. It can be seen that the memory management method provided by the embodiments of this application can greatly improve the memory utilization rate and reduce the peak memory requirement of the computation graph.

[0098] If the above Figures 3 to 5 If the process is executed on the server, the server will also perform the following steps: send the memory address of each of the multiple operators to the terminal device, and the memory address of each operator indicates the memory used by each operator when the terminal device performs data inference using the artificial intelligence model.

[0099] Once the memory address of each operator is determined, during the data inference process using the AI ​​model, the memory address corresponding to each operator can be used to access the memory corresponding to that operator, effectively avoiding data corruption during the inference process.

[0100] The memory management method provided by the embodiments of this application has been described above. The memory management apparatus provided by the embodiments of this application will be described below with reference to the accompanying drawings.

[0101] like Figure 6 As shown, one embodiment of the memory management apparatus 20 provided in this application includes:

[0102] The acquisition unit 201 is used to acquire the computation graph of the artificial intelligence model. The computation graph includes multiple operators and at least one edge, wherein each edge is used to represent the input relationship or output relationship between two operators among the multiple operators.

[0103] The allocation unit 202 is used to allocate memory to each operator in descending order of the memory requirements of each operator among the multiple operators, wherein the memory address of the first operator overlaps with the memory address of the second operator, and the second operator is not a mutex operator of the first operator, wherein the memory of the mutex operator of the first operator cannot be reused by the first operator.

[0104] In this embodiment, when allocating memory for an operator in a computation graph, memory can be allocated from operators that are not mutually exclusive with the operator and have already been allocated memory. This eliminates the need to separately allocate memory for the operator, improving the utilization of already allocated memory and reducing the peak memory requirements of the computation graph. Optionally, the device 20 may further include a determining unit 203, which is used to determine the mutually exclusive operators for each of the plurality of operators.

[0105] Optionally, the mutual exclusion operator includes operators that have an input relationship or an output relationship with the first operator.

[0106] Optionally, the mutual exclusion operator also includes at least one of the following two types of operators: an operator on a second branch in the computation graph that is parallel to the first branch where the first operator is located, and a network output operator that is located before the first operator in the execution order, wherein the network output operator is an operator that has an input relationship with other operators in the computation graph but no output relationship.

[0107] Optionally, the allocation unit 202 is used to allocate memory for the first operator according to the memory requirement of the first operator if the first operator is the operator with the largest memory requirement among multiple operators.

[0108] Optionally, the allocation unit 202 is used to obtain the memory address of each operator whose memory has been allocated if the first operator is not the operator with the largest memory requirement among the multiple operators; allocate memory for the first operator from the memory of the non-mutually exclusive operators whose memory has been allocated, where the non-mutually exclusive operators are operators other than the mutex operators of the first operator among the multiple operators, and the non-mutually exclusive operators include the second operator.

[0109] Optionally, the allocation unit 202 is used to allocate memory corresponding to the memory requirement of the first operator from the remaining memory of the non-mutually exclusive operator if the remaining memory of the non-mutually exclusive operator that has been allocated memory meets the memory requirement of the first operator. The remaining memory of the non-mutually exclusive operator is the total memory allocated to the non-mutually exclusive operator or the memory remaining after the third operator occupies the total memory of the non-mutually exclusive operator. The third operator is an operator that has been allocated memory before the first operator and is not mutually exclusive with the non-mutually exclusive operator.

[0110] Optionally, the allocation unit 202 is further configured to allocate the remaining memory to the first operator if the remaining memory of the non-mutually exclusive operator that has been allocated memory does not meet the memory requirements of the first operator, and then apply for additional spare memory for the first operator. The value of the spare memory is the difference between the memory requirements of the first operator and the remaining memory of the non-mutually exclusive operator.

[0111] Optionally, the sending unit 204 is used to send the memory address of each of the multiple operators to the terminal device, wherein the memory address of each operator indicates the memory used by each operator when the terminal device performs data inference using the artificial intelligence model.

[0112] The memory management device 20 described above can be understood by referring to the corresponding description in the foregoing method embodiment section, and will not be repeated here.

[0113] Figure 7 The diagram shown illustrates a possible logical structure of a computer device 30 according to an embodiment of this application. The computer device 30 includes a processor 301, a communication interface 302, a memory 303, and a bus 304. The processor 301, communication interface 302, and memory 303 are interconnected via the bus 304. In an embodiment of this application, the processor 301 is used to control and manage the operation of the computer device 30; for example, the processor 301 is used to execute... Figures 2 to 5 In the method embodiments, steps 101 to 102, the communication interface 302 is used to support communication by the computer device 30. The memory 303 is used to store the program code and data of the computer device 30.

[0114] The processor 301 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 301 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 304 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0115] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figures 2 to 5 Methods of memory management in China.

[0116] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figures 2 to 5 Methods of memory management in China.

[0117] In another embodiment of this application, a chip system is also provided, the chip system including a processor, the processor being used to support memory management devices to implement the above. Figures 2 to 5 Methods for memory management. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for memory management. This chip system may consist of chips or may include chips and other discrete components.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0119] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0120] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0123] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The above are merely specific implementation methods of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto.

Claims

1. A method of memory management, characterized by, The method comprises: obtaining a computational graph of an artificial intelligence model, the computational graph comprising a plurality of operators and at least one edge, wherein each edge is used to represent an input relationship or an output relationship between two operators in the plurality of operators; allocating memory to each operator in the plurality of operators in turn according to an order from large to small of memory requirements of the each operator, wherein a memory address of a first operator overlaps a memory address of a second operator, and the second operator is not a mutually exclusive operator of the first operator, wherein the mutually exclusive operator of the first operator is an operator mutually exclusive with the first operator in memory usage, and memory of the mutually exclusive operator of the first operator cannot be reused by the first operator.

2. The method of claim 1, wherein, The mutually exclusive operator of the first operator comprises an operator having an input relationship or an output relationship with the first operator.

3. The method of claim 2, wherein, The mutually exclusive operator further comprises at least one of the following two operators: an operator on a second branch parallel to a first branch where the first operator is located in the computational graph, and a network output operator located before the first operator in execution order, the network output operator being an operator having an input relationship with other operators in the computational graph and having no output relationship.

4. The method according to any one of claims 1 to 3, characterized in that, The allocating memory to each operator in the plurality of operators in turn according to an order from large to small of memory requirements of the each operator comprises: if the first operator is an operator with the largest memory requirement in the plurality of operators, then applying memory to the first operator according to the memory requirement of the first operator.

5. The method according to any one of claims 1 to 3, characterized in that, The allocating memory to each operator in the plurality of operators in turn according to an order from large to small of memory requirements of the each operator comprises: if the first operator is not an operator with the largest memory requirement in the plurality of operators, then obtaining a memory address of each operator to which memory has been allocated; allocating memory to the first operator from memory of a non-mutually exclusive operator to which memory has been allocated, the non-mutually exclusive operator being an operator other than the mutually exclusive operator of the first operator in the plurality of operators, the non-mutually exclusive operator comprising the second operator.

6. The method of claim 5, wherein, The allocating memory to the first operator from memory of the non-mutually exclusive operator to which memory has been allocated comprises: if remaining memory of the non-mutually exclusive operator to which memory has been allocated satisfies the memory requirement of the first operator, then allocating memory corresponding to the memory requirement of the first operator from the remaining memory of the non-mutually exclusive operator, the remaining memory of the non-mutually exclusive operator being total memory allocated to the non-mutually exclusive operator or being memory remaining after total memory of the non-mutually exclusive operator is occupied by a third operator, the third operator being an operator allocated memory before the first operator and not mutually exclusive with the non-mutually exclusive operator.

7. The method of claim 6, wherein, The method further comprises: if the remaining memory of the non-mutually exclusive operator to which memory has been allocated does not satisfy the memory requirement of the first operator, then allocating the remaining memory to the first operator, and applying a margin memory to the first operator, the value of the margin memory being a difference between the memory requirement of the first operator and the remaining memory of the non-mutually exclusive operator.

8. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: transmit, to the terminal device, a memory address of each of the plurality of operators, the memory address of the each of the plurality of operators indicating a memory used by the each of the plurality of operators when the terminal device performs data reasoning using the artificial intelligence model.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by one or more processors, implements the method according to any one of claims 1-8.

10. A computing device, comprising: A computer-readable storage medium storing the computer program; The one or more processors are coupled with the computer-readable storage medium, and the computer program, when executed by the one or more processors, implements the method according to any one of claims 1-8.

11. A chip system, characterized by One or more processors are invoked to execute the method according to any one of claims 1-8.

12. A computer program product, characterised in that, A computer program, which, when executed by one or more processors, is used to implement the method according to any one of claims 1-8.

Images