A memory allocation method, a deployment method and related devices

Abstract

The application discloses a memory allocation method, a deployment method and related devices. The method comprises determining the execution order of each operator in the network model, the operator memory, the corresponding dependent operator and the local memory sharing operator; selecting a target operator according to the operator memory and the execution order; and allocating memory for the target operator and the input operator of the target operator according to the dependent operator of the local memory sharing operator of the target operator. The application obtains the dependent operator and the local memory sharing operator of each operator, allocates memory for the operator and the input operator of the operator based on the dependent operator of the local memory sharing operator, and controls the operator that can reuse the memory of the input operator to reuse the memory of the input operator, thereby reducing the required memory resources of the network model. Meanwhile, when the network model is used online, the operator with reused input operator memory is removed from the network model, so that the number of data carrying times in the execution process of the network model is reduced, and the inference speed of the network model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a memory allocation method, a deployment method and related devices. BACKGROUND

[0002] In recent years, the deep learning inference framework based on the mobile terminal has become a key research direction in machine learning, and major manufacturers have successively launched corresponding inference frameworks, such as NCNN, TNN of Tencent, MNN of Alibaba, MDL of Baidu, TFLite of Google and the like. However, due to the limitation of memory resources of embedded devices, the deep learning inference framework applied to the embedded devices is limited by the memory resources, thereby affecting the application of the deep learning inference framework on the embedded devices.

[0003] Therefore, the prior art still needs to be improved and enhanced. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a memory allocation method, a deployment method and related devices in view of the deficiencies of the prior art.

[0005] In order to solve the above technical problem, the first aspect of the embodiment of the present application provides a memory allocation method, which comprises:

[0006] acquiring a preset network model, determining the execution order of each operator in the network model, the operator memory, the corresponding dependent operator and the local memory sharing operator, wherein the dependent operator does not reuse the operator memory, and the local memory sharing operator reuses the operator memory;

[0007] selecting a target operator from the reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order;

[0008] reading the dependent operator of the local memory sharing operator of the target operator;

[0009] allocating memory for the target operator and the input operator of the target operator according to the dependent operator;

[0010] In one implementation manner, the method further comprises: continuing to perform the step of selecting a target operator from the reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order until each operator in the network model is allocated with an operator memory.

[0011] In one implementation manner, the step of allocating memory for the target operator and the input operator of the target operator according to the dependent operator specifically comprises:

[0012] allocating operator memory for the target operator based on the dependent operator when the target operator is a non-prunable operator;

[0013] allocating the same operator memory for the target operator as that of its input operator when the target operator is a prunable operator;

[0014] allocating operator memory for the target operator based on the dependent operator when the target operator is a merge operator, and allocating operator memory for input operators of the target operator in turn according to the execution order.

[0015] In an implementation, the allocating operator memory for the target operator based on the dependent operator specifically includes:

[0016] detecting whether there is a target dependent operator in the dependent operators that has allocated operator memory;

[0017] allocating operator memory for the target operator that does not overlap with that of the target dependent operator when there is a target dependent operator;

[0018] allocating operator memory for the target operator starting from position 0 as the starting position of the target operator when there is no target dependent operator.

[0019] In an implementation, the selecting a target operator from the reference operators of each unallocated operator memory of the network model according to the operator memory and the execution order specifically includes:

[0020] selecting a reference operator with the largest operator memory from the reference operators of each unallocated operator memory of the network model;

[0021] selecting the reference operator with the largest operator memory as the target operator when there are multiple reference operators with the largest operator memory according to the execution order.

[0022] In an implementation, the determining the dependent operator specifically includes:

[0023] selecting a first dependent operator corresponding to each operator in the network model from the operators before and after the operator according to the execution order, to obtain the first dependent operator corresponding to each operator;

[0024] selecting a second dependent operator from the operators after the operator in the network model according to the execution order, wherein the first dependent operator corresponding to the second dependent operator contains the operator;

[0025] The selected first dependent operator and the second dependent operator are taken as the dependent operators corresponding to the operator, to obtain the dependent operators corresponding to each operator.

[0026] In an implementation manner, the selecting the first dependent operator corresponding to the operator from the operators located before the operator in the execution order specifically includes:

[0027] obtaining candidate operators located before the operator in the execution order;

[0028] selecting the first candidate dependent operator corresponding to the operator from the obtained candidate operators based on the operator feature of the operator, wherein the output item of the first candidate dependent operator is the input item of the operator;

[0029] selecting a second candidate dependent operator with unreclaimed operator internal memory from the candidate operators other than the first candidate dependent operator;

[0030] taking the operator, the selected first candidate dependent operator and the second candidate dependent operator as the first dependent operator corresponding to the operator.

[0031] In an implementation manner, the determining process of the local memory sharing operator specifically includes:

[0032] for each operator in the network model, selecting a first local memory sharing operator that multiplexes the operator internal memory of the operator from the operators located after the operator in the execution order;

[0033] selecting a second local memory sharing operator that is multiplexed by the operator internal memory of the operator from the operators located before the operator in the execution order;

[0034] taking the operator, the selected first local memory sharing operator and the second local memory sharing operator as the local memory sharing operator corresponding to the operator, to obtain the local memory sharing operator corresponding to each operator.

[0035] The second aspect of the embodiment of the application provides a memory allocation device, the memory allocation device comprises:

[0036] a determining module, configured to obtain a preset network model, and determine the execution order, the operator internal memory, the corresponding dependent operator and the local memory sharing operator of each operator in the network model, wherein the dependent operator does not multiplex the operator internal memory, and the local memory sharing operator multiplexes the operator internal memory;

[0037] a selecting module, configured to select a target operator from reference operators of each unallocated operator internal memory of the network model according to the operator internal memory and the execution order;

[0038] The reading module is configured to read a dependent operator of the local memory sharing operator of the target operator;

[0039] The allocating module is configured to allocate memory for the target operator and input operators of the target operator according to the dependent operator;

[0040] The executing module is configured to continue the step of selecting a target operator from reference operators of each unallocated operator memory of the network model according to the operator memory and the execution sequence until each operator in the network model is allocated an operator memory.

[0041] The third aspect of the embodiments of the present application provides a network model deployment method, and the method comprises:

[0042] The network model is deployed on a target device, and the memory allocation method described above is used to allocate operator memories for operators of the network model on an offline model;

[0043] Operators in the network model that share input operator memories are removed, and an online inference is performed using the removed network model.

[0044] The fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement steps in the memory allocation method described above, and / or to implement the network model deployment method described above.

[0045] The fifth aspect of the embodiments of the present application provides a terminal device, which comprises a processor, a memory and a communication bus; the memory stores a computer readable program that can be executed by the processor;

[0046] The communication bus realizes connection and communication between the processor and the memory;

[0047] The processor implements steps in the memory allocation method described above and / or implements the network model deployment method described above when the computer readable program is executed.

[0048] Beneficial Effects: Compared with existing technologies, this application provides a memory allocation method, deployment method, and related apparatus. The method includes obtaining a preset network model; determining the execution order, operator memory, corresponding dependent operators, and local memory-sharing operators of each operator in the network model; selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order; reading the dependent operators of the local memory-sharing operators of the target operator; allocating memory for the target operator and its input operators according to the dependent operators; and continuing to execute the step of selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order, until each operator in the network model is allocated operator memory. This application reduces the memory resources required by the network model by obtaining the dependent operators and local memory-sharing operators of each operator, allocating memory for the operators and their input operators based on the dependent operators of the local memory-sharing operators, and controlling operators that can reuse their input operators to reuse their input operator memory. Meanwhile, when using the network model online, operators that reuse the input operator memory are removed from the network model. This reduces the number of data transfers during the network model execution process and improves the inference speed of the network model. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of the memory allocation method provided in this application.

[0051] Figure 2 An example diagram of the network model for the memory allocation method provided in this application.

[0052] Figure 3 for Figure 2 A schematic diagram of the first dependent operator of each operator in the network model shown.

[0053] Figure 4 for Figure 2 The diagram shows the dependent operators of each operator in the network model shown.

[0054] Figure 5 for Figure 2 A schematic diagram of the first local memory sharing operator in the network model shown.

[0055] Figure 6 forFigure 2 A schematic diagram of local memory sharing of operators in the network model shown.

[0056] Figure 7 For Figure 2 A schematic diagram of allocating operator memory for Concat, Conv0, Conv2 and Conv3 in the network model shown.

[0057] Figure 8 For Figure 2 A schematic diagram of allocating operator memory for Conv4 in the network model shown.

[0058] Figure 9 For Figure 2 A schematic diagram of allocating operator memory for Input and Split0 in the network model shown.

[0059] Figure 10 For Figure 2 A schematic diagram of allocating operator memory for Conv1 and Split in the network model shown.

[0060] Figure 11 An example diagram of a network model for the memory allocation method provided by the present application.

[0061] Figure 12 A schematic diagram of allocating operator memory for each operator in the network model shown by the memory allocation method provided by the present application. Figure 11

[0062] Figure 13 A schematic diagram of allocating operator memory for each operator in the network model shown by the existing memory allocation method. Figure 11

[0063] A schematic diagram of the structure of the memory allocation apparatus provided by the present application. Figure 14

[0064] A schematic diagram of the structure of the terminal device provided by the present application. Figure 15 DETAILED DESCRIPTION

[0065] The present application provides a memory allocation method, a deployment method and related apparatuses. To make the purpose, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0066] ​​It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" (or comprise), "comprises" (or comprises) and "comprising" (or comprising) when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. In addition, the use of "connection" or "coupling" herein also includes wireless connection or wireless coupling. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0067] It is to be understood that the terms and phrases used herein have their ordinary meanings as understood by those of ordinary skill in the art in the field of the application, unless otherwise defined. It is further understood that the use of the singular includes the plural unless otherwise defined. It is further understood that the use of relational terms, such as "first," "second," "third," and the like, are used solely to distinguish one from another without necessarily implying an actual order or sequence in time.

[0068] It is to be understood that the sequence of steps in the embodiments and the size of the steps do not mean the order of execution, and the execution order of each process is determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0069] The inventors have found that in recent years, mobile terminal-based deep learning inference frameworks have become a key research direction in machine learning, and major manufacturers have successively launched corresponding inference frameworks, such as Tencent's NCNN, TNN, Ali's MNN, Baidu's MDL, Google's TF Lite, and the like. However, due to the limitation of memory resources of embedded devices, the deep learning inference framework applied to embedded devices will be limited by memory resources, thereby affecting the application of the deep learning inference framework on embedded devices.

[0070] To address the aforementioned issues, this embodiment of the application obtains a preset network model, determines the execution order, operator memory, corresponding dependent operators, and locally shared memory operators for each operator in the network model; selects a target operator from the reference operators of each unallocated operator memory in the network model based on the operator memory and the execution order; reads the dependent operators of the locally shared memory operators of the target operator; allocates memory for the target operator and its input operators based on the dependent operators; and continues executing the step of selecting a target operator from the reference operators of each unallocated operator memory in the network model based on the operator memory and the execution order until each operator in the network model is allocated operator memory. This embodiment of the application reduces the memory resources required by the network model by obtaining the dependent operators and locally shared memory operators of each operator, allocating memory for the operators and their input operators based on the dependent operators of the locally shared memory operators, and controlling operators that can reuse their input operators to reuse their input operator memory. Meanwhile, when using the network model online, operators that reuse the input operator memory are removed from the network model. This reduces the number of data transfers during the network model execution process and improves the inference speed of the network model.

[0071] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0072] This embodiment provides a memory allocation method that can be applied to deep learning inference frameworks to optimize the memory resources required by the framework, reduce the limitations imposed by memory resources, and thus expand the application scope of the deep learning inference framework. For example, after memory optimization, the memory resources of embedded devices can be avoided, allowing the deep learning inference framework to be widely used in embedded devices. In one implementation, the memory allocation method can be used on a network model employing a deep learning inference framework, and the memory allocation method provided in this embodiment can be used to allocate memory operators for each operator in the offline model.

[0073] This embodiment provides a memory allocation method, such as Figure 1 As shown, the method includes:

[0074] S10. Obtain a preset network model and determine the execution order of each operator in the network model, operator memory, corresponding dependent operators, and local memory sharing operators.

[0075] Specifically, the network model is a deep learning inference framework, the network model includes a plurality of operators, each of the plurality of operators corresponds to an execution order, the execution order is used to reflect the execution order of the operator in the inference process of the network model, and the operator is executed earlier if the execution order of the operator is earlier. For example, the network model is the network model as shown in FIG. 1, the execution order of the Input operator in the network model is 1, and the execution order of the Split0 operator is 2. The Input operator is executed before the Split0 operator. The operator memory required by the operator is the memory space required when the operator is executed. For example, the network model is the network model as shown in FIG. 2, the operator memory required by the Input operator in the network model is 40, and the operator memory required by the Conv2D operator is 80. Figure 2 Figure 11

[0076] Dependent operator The dependent operator is determined based on the dependency relationship between the operators. It is assumed that there is an operator A in the network model, the dependent operator of the operator A is determined based on the dependency relationship between the operators, the dependent operator of the operator A is included in the network model, and the dependent operator does not reuse the operator memory of the operator A. It can be understood that when the operator memory of the operator A is allocated to the dependent operator of the operator A, the operator memory of the operator A cannot be allocated to the dependent operator of the operator A. In an implementation, the suggestion principle of the dependency relationship is as follows: 1) the operator A in the network model has a dependency relationship with the operator A; 2) when the input and output of the operator A in the network model cannot share the memory, and the operator A needs to depend on the operator memory of the operator B in the network model when the operator A performs the operator calculation, the operator A and the operator B have a dependency relationship; and 3) the operator A has a dependency relationship with an operator C whose operator memory is not recycled when the operator A performs the calculation.

[0077]

[0078] In an embodiment of the present application, the determination process of the dependent operator of each operator can include the following steps:

[0079] S11, for each operator in the network model, selecting a first dependent operator corresponding to the operator from the operators before the operator and the operator according to the execution order, to obtain the first dependent operator corresponding to each operator;

[0080] S12, for each operator in the network model, selecting a second dependent operator from the operators after the operator according to the execution order, wherein the first dependent operator corresponding to the second dependent operator includes the operator;

[0081] S13, taking the selected first dependent operator and the second dependent operator as the dependent operator of the operator, to obtain the dependent operator corresponding to each operator. ​​​

[0082] Specifically, in the step S11, assuming that the network model comprises an operator A, the first dependent operator corresponding to the operator A is located before the operator A in the execution order, and the operator memory of the operator A depends on the operator memory of the first dependent operator corresponding to the operator A, in other words, the execution process of the operator A itself needs to use the data stored in the operator memory of the first dependent operator dependent on the operator A. For example, as shown in the network model Figure 2 The first dependent operator of the Conv0 operator comprises the Conv0 operator and the Split0 operator, and the operator memory of the Conv0 operator depends on the operator memory of the Conv0 operator and the operator memory of the Split0 operator. It can be understood that, in the execution of the Conv0 operator, the inference process of the Conv0 operator itself needs to use the data stored in the operator memory of the Conv0 operator and the Split0 operator dependent on the Conv0 operator, so that the operator memory of the Conv0 operator and the Split0 operator corresponding to the Conv0 operator cannot be reused.

[0083] In one implementation manner of the embodiment, the first dependent operator corresponding to the operator is selected from the operators before the operator in the execution order and the operator specifically comprises:

[0084] obtaining candidate operators before the operator in the execution order;

[0085] selecting a first candidate dependent operator corresponding to the operator from the obtained candidate operators based on the operator feature of the operator, wherein the output item of the first candidate dependent operator is the input item of the operator;

[0086] selecting a second candidate dependent operator whose operator memory is not recycled from the candidate operators other than the first candidate dependent operator;

[0087] taking the operator, the selected first candidate dependent operator and the second candidate dependent operator as the first dependent operator corresponding to the operator.

[0088] Specifically, assuming that the network model comprises an operator A, the candidate operators of the operator A refer to all operators before the operator A in the execution order, wherein the candidate operators can be empty, 1 or multiple. For example, as shown in the network model Figure 2 The candidate operators of the Input operator are empty, the candidate operators of the Split0 operator are 1, and the candidate operators of the Conv0 operator are 2.

[0089] The operator feature is used to reflect the calculation manner of the operator. The calculation manner can be used to determine whether the input item and the output item of the operator A can share the memory, and further determine the dependent operator of the operator A. When the input item and the output item of the operator A can share the memory, the candidate operator of the output item of the operator A does not need to be dependent on the candidate operator of the operator A when the operator A is executed. When the input item and the output item of the operator A cannot share the memory, the candidate operator of the output item of the operator A needs to be dependent on the candidate operator of the operator A when the operator A is executed. For example, the convolution operator has a dependent relationship with the upper operator of the output convolution operator input item, and the VPU operator and the software operator do not have a dependent relationship with the operator of the output operator input item. That is, after the operator feature of the operator A is obtained, whether the input item and the output item of the operator A can share the memory is detected based on the operator feature, and then the first candidate dependent operator corresponding to the operator A is selected from the candidate operators based on the detection result, so that the operator A and the first candidate dependent operator corresponding thereto cannot share the memory. For example, as shown in the network model of Figure 2 each output item of the Conv0 operator is related to each input item, so that the input item and the output item of the Conv0 operator cannot share the memory, so that the Split0 operator outputting the input item of the Conv0 operator is the first candidate dependent operator of the Conv0 operator.

[0090] The second candidate dependent operator of the operator A refers to the operator memory of the second candidate dependent operator that is not recycled when the operator A is executed. That is, there is an operator C in the operators that are located after the operator A in the execution order, and the output item of the second candidate dependent operator whose operator memory is not recycled is needed when the operator C is executed. It can be understood that the operator memory of the second candidate dependent operator cannot be recycled when the operator A is executed, so that the operator A cannot reuse the operator memory of the second candidate dependent operator. For example, as shown in the network model of Figure 2 When the Conv1 operator is executed, the output item of the Conv0 operator will be used by the Concat operator located after the Conv1 operator, so that the operator memory of the Conv0 operator is not recycled, so that the Conv0 operator is the second candidate dependent operator of the Conv1 operator.

[0091] In addition, after the first candidate dependent operator and the second candidate dependent operator are obtained, since each operator needs to be dependent on its own input item, that is, each operator has a dependent relationship with itself, the operator, the selected first candidate dependent operator and the second candidate dependent operator are used as the first dependent operator of the operator.

[0092] For example, the network model is Figure 2In the network model shown, the input of the Input operator has no input item, and the Input operator only depends on its own output, so the first dependent operator corresponding to the Input operator is the Input operator; the input item and the output item of the Split0 operator can share memory, and the Split0 operator only depends on its own output, so the first dependent operator corresponding to the Split0 operator is the Split0 operator; the input item and the output item of the Conv0 operator cannot share memory, and the Conv0 operator depends on its own output, so the first dependent operator corresponding to the Conv0 operator includes the Split0 operator and the Conv0 operator; the input item and the output item of the Conv1 operator cannot share memory, the Conv1 operator depends on its own output, and the memory of the Conv0 operator cannot be recycled when the Conv1 operator is executed, so the first dependent operator corresponding to the Conv1 operator includes the Split0 operator, the Conv0 operator and the Conv1 operator; by analogy, the first dependent operator corresponding to each operator is as shown in Figure 3 .

[0093] In the step S12, it is assumed that the network model includes an operator A, and the second dependent operator of the operator A is an operator B located after the operator A in the execution order, wherein the operator B after the operator A has established a dependent relationship with the operator A in the execution step S11, that is, the operator A is the first dependent operator of the operator B, so in this step, the operator B is the second dependent operator of the operator A. For example, in the network model shown as Figure 2 , the first dependent operator corresponding to the Conv4 operator includes the Concat operator, so the Conv4 operator is the second dependent operator of the Concat operator.

[0094] In the step S13, after the first dependent operator and the second dependent operator are selected, the selected first dependent operator and the second dependent operator are used as the dependent operators of the operator. In this embodiment, the first dependent operator and each operator located before the operator in the reasoning are established in a dependent relationship, and the second dependent operator and each operator located after the operator in the reasoning are established in a dependent relationship, so that the mutual dependent relationship between each operator in the network model is determined, and the operators having the mutual dependent relationship cannot share memory, so as to allocate continuous memory to each operator based on the mutual dependent relationship. For example, the dependent operators corresponding to each operator in the network model shown as Figure 2 may be as shown in Figure 4 .

[0095] The local memory sharing operator is determined based on a local memory reuse relationship. Assuming that the network model comprises an operator A, the local memory sharing operator of the operator A is determined based on a local memory reuse relationship, and the local memory sharing operator of the operator A is an operator that can reuse the operator memory of the operator A in the network model, in other words, the local memory sharing operator can share the operator memory of the operator A. In an implementation, the determination principle of the local memory reuse relationship can be: 1) the operator A has a local memory reuse relationship with the operator A itself; 2) when an operator B in the network model is connected to the operator A, and the operator B is an operator that can reuse the memory of the output item and the input item (determined according to the operator characteristics of the operator B), the operator A has a local memory reuse relationship with the operator B.

[0096] In an embodiment of the present application, the determination process of the local memory sharing operator corresponding to each operator specifically comprises the following steps:

[0097] For each operator in the network model, a first local memory sharing operator that reuses the operator memory of the operator is selected from the operators located after the operator in the execution order;

[0098] A second local memory sharing operator that is reused by the operator is selected from the operators located before the operator in the execution order;

[0099] The operator, the selected first local memory sharing operator and the second local memory sharing operator are taken as the local memory sharing operator corresponding to the operator, so as to obtain the local memory sharing operator corresponding to each operator.

[0100] Specifically, assuming that the network model comprises an operator A, the first local memory sharing operator of the operator A is located after the operator A in the execution order, and the second local memory sharing operator is located before the operator A in the execution order, wherein the first local memory sharing operator can reuse the operator memory of the operator A, the operator memory of the second local memory sharing operator can be reused by the operator A, and the operator A has a connection relationship with the corresponding first local memory sharing operator and the second local memory sharing operator. That is, the first local memory sharing operator is connected to the operator A and is an output operator of the operator A, and the input item of the first local memory sharing operator reuses the operator memory of the output item of the operator; the second local memory sharing operator is connected to the operator A and is an input operator of the operator A, and the operator memory of the output item of the second local memory sharing operator is reused by the output item of the operator.

[0101] The embodiment obtains the local memory sharing operators of each operator after obtaining the dependent operators corresponding to each operator. In this way, the connection relationship of each operator in the network model can be repeatedly obtained, and the problem of memory allocation error that can be caused by allocating memory to each operator according to the output memory size relationship when only the dependent operators corresponding to each operator are obtained can be avoided. For example, it is assumed that Concat is allocated preferentially, and then memory is allocated to the three input operators corresponding to Concat according to the dependency relationship. Next, memory is allocated to Input, and Input has no dependency relationship with other output memories according to the dependency relationship. Then, memory is allocated to Split0, which reuses the output memory of Input, but also has a dependency relationship with the output memories of Conv0 and Conv1, resulting in memory allocation error.

[0102] For example, as shown in a network model in Figure 2 , each operator in the network model has its own local memory sharing operator. The Input operator is connected to the Split0 operator, and the Split0 operator is an input-output memory reuse operator. Therefore, the Split0 operator is the first local memory sharing operator of the Input operator. Then, the connection is made to the lower layer. Since the output item of the Split0 operator reuses the operator memory of the output item of the Input operator, the connection operators of the Split0 operator can be regarded as the connection operators of the Input operator, that is, Conv0 and Conv1 are the connection operators of the Input operator. However, Conv0 and Conv1 judge that the output item does not reuse the input item according to the operator characteristics, so Conv0 and Conv1 are not the first local operator of the Input operator. In addition, the connection operators of Conv0 and Conv1 cannot be regarded as the connection operators of the Input operator. Therefore, the first local memory sharing operator of the Input operator is the Split0 operator. By analogy, Figure 2 , the first local memory sharing operator of each operator in the network model shown in Figure 5 may be obtained. After the first local memory sharing operator is obtained, the second local memory sharing operator of each operator needs to be obtained by traversing in reverse according to the inference order. For example, Conv4 is connected to Concat, and Concat is not an operator that reuses the input item memory of the output item. Therefore, Concat is not the second local memory sharing operator of Conv4. The operators connected to Concat, that is, Conv0, Conv2, and Conv3, are all the second local memory sharing operators of Concat according to the memory reuse determination principle. After the reverse traversal is completed, Figure 2 , the local memory sharing operator corresponding to each operator in the network model shown in Figure 6 may be obtained.

[0103] S20, selecting a target operator from the reference operators of the unallocated operator memories of the network model according to the operator memories and the execution order.

[0104] Specifically, the reference operator is an operator in the network model whose operator memory is unallocated, and the target operator is an operator in the network model whose operator memory is the largest among the reference operators. For example, as shown in the network model in Figure 2 Fig. 2, when the operator memories of all operators in the network model are unallocated, the operator memory of the Concat operator is the largest, and the Concat operator is the target operator. In addition, in actual applications, there can be operators in the network model whose operator memories are the same, so that when the target operator is selected from the network model, multiple target operators can be selected. Based on this, the selection of the target operator from the reference operators of the unallocated operator memories of the network model according to the operator memories and the execution order is specifically as follows:

[0105] selecting a reference operator whose operator memory is the largest from the reference operators of the unallocated operator memories of the network model;

[0106] When the reference operator whose operator memory is the largest is multiple, the reference operator whose operator memory is the largest at the front is selected as the target operator according to the execution order.

[0107] Specifically, the reference operator is an operator in the network model whose operator memory is unallocated, and the target operator is an operator in the network model whose operator memory is the largest among the reference operators. For example, as shown in the network model in Figure 2 Fig. 2, when the operator memories of all operators in the network model are unallocated, the operator memory of the Concat operator is the largest, and the Concat operator is the target operator. In addition, in actual applications, there can be operators in the network model whose operator memories are the same, so that when the target operator is selected from the network model, multiple target operators can be selected. Based on this, the selection of the target operator from the reference operators of the unallocated operator memories of the network model according to the operator memories and the execution order is specifically as follows:

[0108] S30, reading the dependent operators of the local memory sharing operators of the target operator.

[0109] Specifically, the local memory sharing operators of the target operator and the dependent operators of the local memory sharing operators are obtained through the above steps, that is, after the target operator is selected, the local memory sharing operators corresponding to the target operator can be directly read, and then the dependent operators corresponding to each local memory sharing operator are read. The dependent operators of the local memory sharing operators are all the dependent operators relied on by the memory allocation of the target operator, because the local memory sharing operators reuse the operator memory of the target operator, while the dependent operators of the local memory sharing operators do not reuse the operator memory of the local memory sharing operators, so the operator memory of the target operator cannot overlap with the operator memory of the dependent operators corresponding to the target operator and the dependent operators of the local memory sharing operators corresponding to the target operator. In addition, because the target operator is its own local sharing operator, when the dependent operators of the local sharing operators corresponding to the target operator are obtained, the dependent operators of the target operator itself are also obtained. Therefore, when allocating the operator memory for the target operator, only the operator memory of each local memory sharing operator corresponding to the target operator needs to be obtained to obtain all the dependent operators affecting the allocation of the operator memory of the target operator.

[0110] S40, allocating memory for the target operator and the input operators of the target operator according to the dependent operators.

[0111] Specifically, the input operators are connected with the target operator, and the output items of the input operators are the input items of the target operator. When the dependent operators are obtained, because the operator memory of the target operator does not reuse the operator memory of the dependent operators, when the operator memory is allocated for the target operator, the operator memory allocated for the target operator does not overlap with the operator memory of the dependent operators. In addition, the operator characteristics of each operator in the network model are different, and when the memory is allocated for the target operator and the input operators of the target operator based on the dependent operators, the memory can be directly allocated for the target operator, or the memory can be allocated for the target operator and the input operators of the target operator.

[0112] Based on this, in one implementation, the step S40: allocating memory for the target operator and the input operators of the target operator according to the dependent operators specifically includes:

[0113] S41, when the target operator is a non-removable operator, allocating the operator memory for the target operator based on the dependent operators;

[0114] S42, when the target operator is a removable operator, allocating the same operator memory for the target operator as the operator memory of the input operators of the target operator;

[0115] S43, when the target operator is a merge operator, allocating operator memory for the target operator based on the dependent operator, and sequentially allocating operator memory for input operators of the target operator in execution order

[0116] Specifically, in the step S41, the non-prunable operator is an operator that performs operator computation by itself, wherein the non-prunable operator can be determined based on an operator characteristic or according to a pre-set prunable operator set. When determined based on the operator characteristic, the operator characteristic of the target operator is acquired when the target operator is acquired, and the target operator is a non-prunable operator when the operator characteristic is to perform operator computation by itself. When determined according to the pre-set prunable operator set, whether the target operator is included in the prunable operator set is detected when the target operator is acquired, and the target operator is a non-prunable operator when not included in the prunable operator set and not a merge operator.

[0117] In one typical implementation, since the preset network model is known, each operator included in the network model can be acquired in advance before the memory is allocated for the network model, and an operator type is allocated for each operator, wherein the operator type includes a prunable operator, a non-prunable operator, and a merge operator. Then, when the memory is allocated for the operator, the operator type carried by the operator can be directly read, and the memory is allocated for each operator based on the extracted operator type, which can avoid the operator type determination process in the memory allocation process, and can improve the memory allocation speed. On the other hand, it can also avoid the problem of memory allocation error caused by incorrect operator type determination.

[0118] In one implementation of the embodiment, the allocating operator memory for the target operator based on the dependent operator specifically includes:

[0119] detecting whether there is a target dependent operator in the dependent operators that has allocated operator memory;

[0120] when there is the target dependent operator, allocating operator memory for the target operator that does not overlap with the operator memory of the target dependent operator;

[0121] when there is no target dependent operator, taking position 0 as the starting position of the target operator, and allocating operator memory for the target operator from the starting position.

[0122] Specifically, the dependent operator is a dependent operator of a local memory sharing operator corresponding to the target operator, wherein the dependent operator can have allocated operator memory or can not have allocated operator memory. For example, the target operator is an operator that performs computation on a local memory shared by a plurality of operators, and the dependent operator is an operator that performs computation on the local memory shared by the plurality of operators. Figure 2The Concat operator in the network model shown corresponds to the obtained dependent operators including the Concat operator, the Conv3 operator, the Conv2 operator, and the Conv0 operator, and the Concat operator is as shown in Figure 2 The largest operator in the network model shown is the operator, so when allocating memory for the Concat operator, the Concat operator, the Conv3 operator, the Conv2 operator, and the Conv0 operator are not allocated operator memory, and there is no target dependent operator in the Concat operator corresponding to the obtained dependent operators; the target operator is the Conv4 operator in the network model shown, and the Conv4 operator corresponding to the obtained dependent operators includes the Concat operator, the Concat operator has been allocated operator memory, and the target dependent operator Concat operator exists in the Conv4 operator corresponding to the obtained dependent operators. Figure 2

[0123] When there is a target dependent operator, the target operator does not reuse the operator memory of the target dependent operator, so the operator memory of the target operator cannot overlap with the operator memory of the target dependent operator. Therefore, the operator position of the target operator can be set after the operator memory of the target dependent operator. In an implementation, when there is a target dependent operator, the position adjacent to and after the target dependent operator is used as the starting position of the target operator, and the operator memory of the target operator is allocated from the starting position. In this way, using the position adjacent to and after the target dependent operator as the starting position of the target operator can make the network model occupy as little memory resource as possible. Of course, in actual applications, any position of the memory resource after the memory resource occupied by the operator memory of the target dependent operator can be used to allocate operator memory for the target operator, and the like. In addition, when there is no target dependent operator, the operator memory of the target operator memory can be allocated to any position, and in this case, position 0 can be used as the starting position of the target operator to allocate operator instructions for the target operator. Of course, in actual applications, other positions can also be used as the starting position, for example, position 1, position 2, and the like.

[0124] In the step S42, the target operator can be a deletable operator that can reuse the operator memory of its input operator, and when the target operator is a deletable operator, the target operator can reuse the operator memory of its input operator, so the same operator memory as the input operator is allocated, wherein the output item of the input operator is the input item of the target operator. For example, as shown in Figure 2 ​The Split0 operator in the network model shown is a cullable operator, and the input operator of the Split0 operator is the Input operator, so the memory of the Split0 operator is the same as that of the Input operator. In addition, it is worth noting that since a cullable operator can reuse the operator memory of its input operator, the operator memory size of the cullable operator is the same as that of its input operator; and in this embodiment, when the operator memory sizes are the same, the memory is allocated to the operators with the same operator memory size according to the execution order, and the execution order of the input operator of the target operator is earlier than that of the target operator. Thus, when the memory is allocated to the target operator, the input operator of the target operator has already been allocated operator memory, so the operator memory of the target operator can be allocated the same as that of its input operator.

[0125] In the step S43, the merge operator is an operator for performing a merge operation, and the merge operator itself does not perform operator execution, but only performs data merging of input items, so that the merge operator can also be culled when performing operator culling. However, when the merge operator is culled, it is necessary to ensure that the operator memory of each input operator of the merge operator is continuous, so that the operator with the output item of the merge operator as input can correctly obtain the output items of each input operator of the merge operator. Thus, when the target operator is a merge operator, after obtaining the local memory sharing operator of the target operator and allocating operator memory to the target operator based on the dependent operator, operator memory is also allocated to the input operators corresponding to the merge operator, wherein the operator memory allocated to the input operators corresponding to the merge operator can overlap with the operator memory of the merge operator, but the operator memory between the input operators cannot overlap. Wherein, the process of obtaining the local memory sharing operator of the target operator and allocating operator memory to the target operator based on the dependent operator is the same as that in step S41, which will not be repeated here, and can be referred to the description of step S41.

[0126] In an embodiment of the present application, the method further comprises:

[0127] S50, continue to perform the step of selecting a target operator from the reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order until each operator in the network model is allocated operator memory.

[0128] Specifically, before the step of selecting the target operator with the largest operator memory from the reference operators of the network model is continued to be executed, it can be detected whether there is an operator without allocated operator memory in the network model; when there is an operator without allocated operator memory, the step of selecting the target operator with the largest operator memory from the reference operators of the network model is continued to be executed; when there is no operator without allocated operator memory, it is indicated that the memory allocation of the network model has been completed, and the step of selecting the target operator with the largest operator memory from the reference operators of the network model is stopped to be executed.

[0129] To further illustrate the memory allocation method provided by the embodiment, a specific example is given below to specifically illustrate the memory allocation method provided by the embodiment.

[0130] The preset network model is as shown in Figure 2 The reference operator corresponding to each operator in the network model is as shown in Figure 4 The local memory sharing operator corresponding to each operator is as shown in Figure 6 The memory allocation process of each operator in the network model is as follows:

[0131] The Concat is allocated memory, the Concat is a merging operator, the dependent operator corresponding to the Concat and the dependent operator corresponding to the local memory sharing operator corresponding to the Concat include the Concat, the Conv3, the Conv2 and the Conv0, and the Concat, the Conv3, the Conv2 and the Conv0 are all not allocated operator memory, so the Concat is not limited by memory and can be placed from the Offset=0 position; the input operators of the Concat include the Conv3, the Conv2 and the Conv0, after the input operators of the Concat are allocated memory, the operator memory is allocated to the Conv3, the Conv2 and the Conv0 in sequence, as shown in Figure 7

[0132] The Conv4 is allocated memory, the dependent operator corresponding to the Conv4 and the dependent operator corresponding to the local memory sharing operator corresponding to the Conv4 include the Conv4 and the Concat, the Concat has been allocated operator memory, so a piece of memory capable of storing the output of the Conv4 and not overlapping with the memory of the Concat is found from the offset=0 position, as shown in Figure 8

[0133] ​​The operator memory of Input, Split0, Conv1 and Split1 is the same, and the memory is allocated for Input first according to the execution order. The dependent operators corresponding to Input and the dependent operators of the local memory sharing algorithm corresponding to Input include Input, Split0, Conv0 and Conv1. The operator memory of Conv0 has been allocated. The Input operator starts from the position of offset = 0 to find a piece of memory that can store the output of Input and does not overlap with the memory of Conv0. Figure 9

[0134] The memory of Conv1 is allocated. The dependent operators corresponding to Conv1 and the dependent operators of the local memory sharing algorithm corresponding to Conv1 include Split0, Conv0, Conv2 and Conv3. The memory of Conv1 starts from the position of offset = 0 to find a piece of memory that can store the output of Conv1 and does not overlap with the memory of Split0, Conv0, Conv2 and Conv3. The memory of Split1 is allocated and is the same as the memory of the input operator Conv1, as shown in Figure 10 .

[0135] In order to further illustrate the effect of the memory allocation method provided in the embodiment, the network model as shown in Figure 11 is subjected to memory allocation by using the memory allocation method provided in the embodiment and the existing memory allocation method (Greedy by Size for Offset Calculation method), respectively. The memory usage of the memory allocation method of the embodiment is as shown in Figure 12 , and the memory usage of the existing memory allocation method is as shown in Figure 13 . As can be seen from Figure 12 and Figure 13 , the memory size used by the memory method provided in the embodiment is 160, and the memory size used by the existing memory allocation method is 180. The memory allocation method provided in the embodiment can use better memory resources.

[0136] ​In summary, this embodiment provides a memory allocation method, which includes obtaining a preset network model; determining the execution order, operator memory, corresponding dependent operators, and local memory-sharing operators of each operator in the network model; selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order; reading the dependent operators of the local memory-sharing operators of the target operator; allocating memory for the target operator and its input operators according to the dependent operators; and continuing to execute the step of selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order, until each operator in the network model is allocated operator memory. This application reduces the memory resources required by the network model by obtaining the dependent operators and local memory-sharing operators of each operator, allocating memory for the operators and their input operators based on the dependent operators of the local memory-sharing operators, and controlling operators that can reuse their input operators to reuse their input operator memory. Meanwhile, when using the network model online, operators that reuse the input operator memory are removed from the network model. This reduces the number of data transfers during the network model execution process and improves the inference speed of the network model.

[0137] Based on the above memory allocation method, this embodiment provides a method for deploying a network model, the method comprising:

[0138] The network model is deployed on the target device, and operator memory is allocated to each operator of the network model on the offline model using the memory allocation method described above;

[0139] The operators that reuse input operators in the network model are removed from the operator memory, and the removed network model is used for online inference.

[0140] Specifically, the operators used for reusing input operators are the culling and merging operators in the network model. After removing the operators from the operator memory of the reusing input operators, the input operators corresponding to the removed operators can be connected to the output operators corresponding to the removed operators. In this way, when the network model performs online inference, the output operators corresponding to the removed operators can directly use the operators in the operator memory of the input operators corresponding to the removed operators. This reduces data transfer during the execution of the network model and improves the inference speed of the network model.

[0141] Based on the above memory allocation method, this embodiment provides a memory allocation device, such as... Figure 14 As shown, the memory allocation device includes:

[0142] The determining module 100 acquires a preset network model, determines an execution order of each operator in the network model, an operator memory, a corresponding dependent operator and a local memory sharing operator, wherein the dependent operator does not reuse the operator memory, and the local memory sharing operator reuses the operator memory.

[0143] The selecting module 200 is configured to select a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order.

[0144] The reading module 300 is configured to read a dependent operator of the local memory sharing operator of the target operator.

[0145] The allocating module 400 is configured to allocate memory for the target operator and an input operator of the target operator according to the dependent operator.

[0146] The executing module 500 is configured to continue to execute the step of selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order until each operator in the network model is allocated an operator memory.

[0147] Based on the above memory allocation method, the embodiment provides a computer readable storage medium, the computer readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the memory allocation method.

[0148] Based on the above memory allocation method, the embodiment further provides a terminal device, as shown in the figure. Figure 15 The terminal device includes at least one processor 20, a display screen 21, a memory 22, and a communications interface 23, and a bus 24. The processor 20, the display screen 21, the memory 22 and the communications interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in an initial setting mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0149] In addition, the logic instructions in the memory 22 can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0150] The memory 22, as a computer readable storage medium, can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the method in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions or modules stored in the memory 22, thereby performing functional applications and data processing, that is, implementing the method in the above embodiments.

[0151] The memory 22 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory. For example, a variety of media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., can also be a transitory storage medium.

[0152] In addition, the specific process of the memory medium and the plurality of instructions in the terminal device loaded and executed by the processor has been described in detail in the above method, and will not be described one by one here.

[0153] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A memory allocation method characterized by comprising: The method comprises: obtaining a preset network model, determining execution order of each operator in the network model, operator memory, corresponding dependent operators and local memory sharing operators, wherein the dependent operators do not reuse the operator memory, and the local memory sharing operators reuse the operator memory; selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order; reading dependent operators of the local memory sharing operators of the target operator; allocating memory for the target operator and input operators of the target operator according to the dependent operators; allocating memory for the target operator and input operators of the target operator according to the dependent operators specifically comprises: allocating operator memory for the target operator based on the dependent operators when the target operator is a non-removable operator; allocating the same operator memory as input operators of the target operator for the target operator when the target operator is a removable operator; allocating operator memory for the target operator based on the dependent operators and sequentially allocating operator memory for input operators of the target operator according to the execution order when the target operator is a merged operator; allocating operator memory for the target operator based on the dependent operators specifically comprises: detecting whether there is a target dependent operator with allocated operator memory in the dependent operators; allocating operator memory for the target operator which does not overlap with operator memory of the target dependent operator when the target dependent operator exists; allocating operator memory for the target operator starting from position 0 as a starting position of the target operator when the target dependent operator does not exist.

2. The memory allocation method of claim 1, wherein, The method further comprises: continuing to perform the step of selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order until each operator in the network model is allocated operator memory.

3. The memory allocation method of claim 1, wherein, selecting a target operator from reference operators of each unallocated operator memory in the network model according to the operator memory and the execution order specifically comprises: selecting a reference operator with the largest operator memory from reference operators of each unallocated operator memory in the network model; selecting a reference operator with the largest operator memory located at the front as a target operator according to the execution order when there are multiple reference operators with the largest operator memory.

4. The memory allocation method of claim 1, wherein, The determination process of the dependent operators specifically comprises: selecting a first dependent operator corresponding to each operator from operators located before the each operator and the each operator according to the execution order to obtain first dependent operators corresponding to each operator for each operator in the network model; selecting a second dependent operator from operators located after the each operator according to the execution order, wherein the first dependent operator corresponding to the second dependent operator contains the each operator for each operator in the network model; selecting the first dependent operator and the second dependent operator as the dependent operator corresponding to the each operator to obtain dependent operators corresponding to each operator.

5. The memory allocation method of claim 4, wherein, The first dependent operator corresponding to each operator in the network model is selected from the operators before the each operator and the each operator in the execution order, and specifically comprises: obtaining a candidate operator before the each operator in the execution order; selecting a first candidate dependent operator corresponding to the each operator from the obtained candidate operators based on the operator characteristics of the each operator, wherein the output item of the first candidate dependent operator is the input item of the each operator; selecting a second candidate dependent operator with unreclaimed operator internal memory from the candidate operators other than the first candidate dependent operator; taking the each operator, the selected first candidate dependent operator and the second candidate dependent operator as the first dependent operator corresponding to the each operator.

6. The memory allocation method of claim 1, wherein, The determination process of the local memory sharing operator specifically comprises: for each operator in the network model, selecting a first local memory sharing operator that reuses the operator internal memory of the each operator from the operators after the each operator in the execution order; selecting a second local memory sharing operator that is reused by the each operator from the operators before the each operator in the execution order; taking the each operator, the selected first local memory sharing operator and the second local memory sharing operator as the local memory sharing operator corresponding to the each operator to obtain the local memory sharing operator corresponding to each operator.

7. A memory allocation apparatus for implementing the steps of the memory allocation method according to any one of claims 1 to 6, characterized by The memory allocation device comprises: a determination module configured to obtain a preset network model, and determine the execution order of each operator in the network model, operator internal memory, corresponding dependent operator and local memory sharing operator, wherein the dependent operator does not reuse the operator internal memory, and the local memory sharing operator reuses the operator internal memory; a selection module configured to select a target operator from reference operators of each unallocated operator internal memory in the network model according to the operator internal memory and the execution order; a reading module configured to read the dependent operator of the local memory sharing operator of the target operator; an allocation module configured to allocate memory for the target operator and an input operator of the target operator according to the dependent operator; an execution module configured to continue the step of selecting a target operator from reference operators of each unallocated operator internal memory in the network model according to the operator internal memory and the execution order until each operator in the network model is allocated with operator internal memory.

8. A method of deploying a network model, the method comprising: The method comprises: deploying a network model on a target device, and allocating operator internal memory for each operator of the network model on an offline model by using the memory allocation method in any one of claims 1-6; removing an operator that reuses the operator internal memory of an input operator in the network model, and performing online inference by using the removed network model.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the memory allocation method in any one of claims 1-6, and / or to implement the deployment method of the network model in claim 8.

10. A terminal device, comprising: comprising: a processor, a memory and a communication bus; the memory stores a computer readable program which can be executed by the processor; the communication bus realizes the connection communication between the processor and the memory; the processor realizes the steps in the memory allocation method as claimed in any one of claims 1-6 and / or realizes the deployment method of the network model as claimed in claim 8 when executing the computer readable program.

Images