Memory allocation method, device, equipment, and storage medium

Abstract

The present application discloses a memory allocation method and apparatus, a device, and a storage medium; wherein the method comprises: defining a rectangular block of a computing node of a neural network based on the memory usage of the computing node and the time information of the memory being occupied; wherein the side length of the rectangular block in a first dimension represents the memory usage, and the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension; stacking and arranging the rectangular blocks of each computing node of the neural network in the first dimension, so as to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition; and allocating corresponding memory addresses to each computing node of the neural network based on the target arrangement result.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and is related to but not limited to memory allocation methods and devices, equipment, and storage media. Background Art

[0002] During the training or reasoning process of a neural network, each operator involves data input and output; therefore, the system's storage space needs to provide corresponding memory space for caching this data. As the performance of neural networks improves, the amount of data they process also grows exponentially. Therefore, how to improve the efficiency of the system's storage space is an issue that every neural network training and reasoning system needs to consider. Summary of the invention

[0003] In view of this, the memory allocation method, apparatus, device, and storage medium provided in the present application can reduce the memory overhead at the same time while improving the memory utilization rate during the training or reasoning process of the neural network.

[0004] According to one aspect of an embodiment of the present application, a memory allocation method is provided, comprising: defining a rectangular block of a computing node of a neural network based on the memory usage of the computing node and the time information of the memory being occupied; wherein the side length of the rectangular block in a first dimension represents the memory usage, and the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension; stacking and arranging the rectangular blocks of each computing node of the neural network in the first dimension, so as to arrange a target arrangement result in which the overall length in the first dimension satisfies a first condition; and allocating corresponding memory addresses to each computing node of the neural network based on the target arrangement result.

[0005] According to one aspect of an embodiment of the present application, a memory allocation device is provided, comprising: a definition module, configured to define a rectangular block of a computing node of a neural network based on the memory usage of the computing node and the time information of the memory occupation; wherein the side length of the rectangular block in a first dimension represents the memory usage, and the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension; an arrangement module, configured to stack and arrange the rectangular blocks of each computing node of the neural network in the first dimension, so as to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition; an allocation module, configured to allocate corresponding memory addresses to each computing node of the neural network based on the target arrangement result.

[0006] According to one aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0007] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0008] In an embodiment of the present application, the memory allocation problem of the neural network is converted into an optimization problem similar to that of Tetris. The target arrangement result is found in the dimension of memory usage (i.e., the first dimension) in which the overall length of each rectangular block stacked satisfies the first condition, and the corresponding memory space is allocated to each computing node based on the result. In this way, the memory overhead of the neural network at the same time is reduced while improving the memory utilization rate.

[0009] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0012] Figure 1 A schematic diagram of the implementation flow of the memory allocation method provided in the embodiment of the present application;

[0013] Figure 2 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0014] Figure 3 A two-dimensional schematic diagram of the time and memory size of the memory occupied by the computing node provided in the embodiment of the present application;

[0015] Figure 4 A schematic diagram of the implementation flow of the memory allocation method according to an embodiment of the present application;

[0016] Figure 5 A schematic diagram of a first arrangement result provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of a second arrangement result provided in an embodiment of the present application;

[0018] Figure 7 A schematic diagram of the implementation flow of the memory allocation method according to an embodiment of the present application;

[0019] Figure 8 A schematic diagram of a third arrangement result provided in an embodiment of the present application;

[0020] Fig. 9 A two-dimensional schematic diagram of the time and memory size of the memory occupied by the computing node provided in the embodiment of the present application;

[0021] Fig.10 A two-dimensional schematic diagram of the time and memory size of the memory occupied by the computing node provided in the embodiment of the present application;

[0022] Fig.11 A schematic diagram of converting a neural network memory allocation problem into a Tetris problem provided in an embodiment of the present application;

[0023] Fig.12 This is a schematic diagram of the structure of the memory allocation device according to an embodiment of the present application;

[0024] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the specific technical scheme of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application do not represent a specific ordering of objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0029] The embodiment of the present application provides a memory allocation method, which is applied to an electronic device. The electronic device can be various types of devices with information processing capabilities during implementation. For example, the electronic device can include any device that performs neural network training or reasoning, such as a mobile phone, a tablet computer, a laptop computer, or a computer. The function implemented by the method can be implemented by calling program code by a processor in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0030] Figure 1 A schematic diagram of the implementation flow of the memory allocation method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the method may include the following steps 101 to 103:

[0031] Step 101, based on the memory usage of the computing nodes of the neural network and the time information of the memory occupation, define a rectangular block of the computing nodes; wherein the side length of the rectangular block in a first dimension represents the memory usage, the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension.

[0032] It is understandable that after the structure of the neural network is fixed, the memory demand is determined in terms of size and time. Therefore, the memory allocation of a neural network can be represented as rectangular blocks based on the memory usage and the time information of the memory occupied, and the optimal solution for memory allocation is solved based on the rectangular blocks. For ease of understanding, for example, Figure 2 Taking the neural network shown in the figure as an example, each circle in the figure represents a computing node of the neural network. The number above the circle represents the order in which the computing node participates in the calculation according to the structure of the neural network; the number below the circle represents the memory size required for the data generated by the operation of the computing node. Among them, the first computing node represents DMA Load, and the eighth computing node represents DMA Store. The corresponding operations are moving data in and out of the external storage device.

[0033] based on Figure 2 The memory usage of the neural network shown in Figure 2 can be obtained by Figure 3The time-memory size two-dimensional graph shown is reflected. Among them, each rectangular block corresponds to a computing node of the neural network, the horizontal axis is the second dimension, representing time; the vertical axis is the first dimension, representing memory usage. It is easy to understand that the side length of the rectangular block on the horizontal axis represents the time information of the computing node occupying the memory, and the side length of the rectangular block on the vertical axis represents the memory usage of the computing node. It can be seen that the memory allocation problem of the neural network is transformed into an optimization problem similar to that of Tetris. Based on this, the order of appearance of each rectangular block can be adjusted in the vertical axis direction, so as to arrange the arrangement result in which the height of the highest point (i.e., the overall length) of all the rectangular blocks stacked together meets the first condition. That is, after defining the rectangular blocks of each computing node, execute step 102.

[0034] Step 102: stack and arrange the rectangular blocks of the computing nodes of the neural network in the first dimension, so as to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition.

[0035] It should be noted that in this application, there is no restriction on the first condition, and it can be set flexibly. For example, the first condition is that the overall length is the minimum; for another example, the first condition is that the overall length is less than the length threshold. In short, as long as the memory overhead (that is, memory usage) at the same time can be arranged to meet the memory overhead requirement.

[0036] It can be understood that when the rectangular blocks are stacked and arranged in the first dimension, the overall length is the smallest, which means that the memory overhead at the same time is the smallest when performing training or reasoning of the neural network. In other words, the memory overhead at the same time is the smallest under the premise of ensuring the normal operation of the neural network.

[0037] In some embodiments, the electronic device may implement step 102 through steps 403 to 406 of the following embodiments; in other embodiments, the electronic device may also implement step 102 through steps 703 to 705 of the following embodiments.

[0038] Step 103: Based on the target arrangement result, a corresponding memory address is allocated to each computing node of the neural network.

[0039] In an embodiment of the present application, the memory allocation problem of the neural network is converted into an optimization problem similar to that of Tetris. The target arrangement result is found in the dimension of memory usage (i.e., the first dimension) in which the overall length of each rectangular block stacked satisfies the first condition, and the corresponding memory space is allocated to each computing node based on the result. In this way, the memory overhead of the neural network at the same time is reduced while improving the memory utilization rate.

[0040] In some embodiments, before executing step 101, the method further includes: determining the memory usage and the time information of the memory occupied by each computing node of the neural network during operation.

[0041] It is understandable that, since the structure of the neural network is fixed before the calculation, the storage space required for the input and output data of each operator and the time of their occurrence can be known exactly before the neural network is trained or inferred. Therefore, the memory usage and the time information of the memory occupied by each computing node of the neural network during operation can be determined in advance, and the optimal solution for memory allocation can be solved based on this. In some embodiments, the time information includes the start time and end time of the expected memory occupation.

[0042] It should be noted that one computing node can correspond to one operator, and one computing node can also correspond to multiple operators, which is not limited here.

[0043] The present application embodiment further provides a memory allocation method, Figure 4 Schematic diagram of the implementation process of the memory allocation method of the embodiment of the present application, such as Figure 4 As shown, the method includes steps 401 to 407:

[0044] Step 401, determining the memory usage and memory occupation time information of each computing node of the neural network during operation;

[0045] Step 402: Based on the memory usage of the computing node and the time information of the memory occupation, define a rectangular block of the computing node; wherein the side length of the rectangular block in a first dimension represents the memory usage, the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension;

[0046] Step 403 : Evaluate the contribution parameter of each rectangular block to be arranged in a stacked arrangement in the first dimension to reducing the overall length of the target arrangement result.

[0047] It can be understood that the goal of this application is to find the optimal solution for the stacking arrangement of each rectangular block, that is, the memory overhead of the neural network at the same time during training or inference is minimal or less than the memory usage threshold (i.e., length threshold). Figure 3 For example, it can be seen that the lower the rectangular block is placed, the better, and the closer the rectangular blocks are to each other, the better. In this application, there is no restriction on the factor parameters used to determine the contribution parameters of the rectangular blocks. These factor parameters include but are not limited to at least one of the following:

[0048] (1) The number of rectangular blocks that are in contact with the rectangular block when the rectangular blocks are stacked and arranged in the first dimension. It is easy to understand that the more rectangular blocks that are in contact with the rectangular block, the more times the same memory space is reused, that is, the higher the memory utilization rate.

[0049] (2) When the rectangular blocks are stacked and arranged in the first dimension, the area composed of the entire stacked rectangular blocks has the maximum first length in the first dimension. Figure 3 For example, it is easy to understand that the first length is the vertical distance from the highest point of the area composed of the currently arranged rectangular blocks to the horizontal axis.

[0050] Taking into account both the need to ensure that the overall length of the target arrangement result finally obtained in the first dimension meets the requirements, that is, the memory overhead at the same time meets the index requirements, and the need to simplify the complexity of the algorithm, in some embodiments, the maximum first length of the first arrangement result (that is, the overall length of the first arrangement result in the first dimension) formed by each of the rectangular blocks to be arranged when stacked and arranged in the first dimension and the currently determined arrangement result is determined separately, and the first length is used as the contribution parameter of the corresponding rectangular block.

[0051] It can be understood that, unlike the falling blocks in the Russian game, in this application, since the time that the computing nodes corresponding to the rectangular blocks occupy the memory is fixed, the arrangement order of each rectangular block in the first dimension can only be adjusted. Based on this, it is necessary to determine the maximum first length of the first arrangement result composed of the determined arrangement results when each rectangular block to be arranged is stacked and arranged in the first dimension one by one. For ease of understanding, Figure 3 Assume that the currently determined arrangement result is as follows: Figure 5 As shown in 501, the rectangular block corresponding to the computing node No. 0 (referred to as rectangular block No. 0), the rectangular blocks to be arranged include rectangular blocks No. 1 to No. 7; then according to the above method, the first arrangement result that can be obtained is as follows Figure 5 As shown in 502 to 508, the first lengths corresponding to the rectangular blocks can be obtained as shown in Table 1:

[0052] Table 1

[0053] Rectangular blocks to be arranged First length (bytes) Rectangular blocks to be arranged First length (bytes) Rectangular block No. 1 60 No. 5 rectangular block 64 Rectangular block No. 2 36 No. 6 rectangular block 32 Rectangular block No. 3 32 No. 7 rectangular block 40 No. 4 rectangular block 32

[0054] Step 404: determine a target rectangular block based on the rectangular blocks to be arranged whose contribution parameters satisfy the second condition.

[0055] The factor parameters referenced in calculating the contribution parameter are different, and accordingly, the second condition is also different.

[0056] For example:

[0057] When the factor parameters referenced in calculating the contribution parameter are multiple factor parameters, for example, the reference factor parameters are the above-mentioned factor parameter (1) and factor parameter (2), then the contribution parameter = factor parameter (1) - factor parameter (2), and the corresponding second condition is greater than a specific threshold. For example, when a rectangular block is stacked and arranged in the first dimension, the number of rectangular blocks that contact the rectangular block is 3, and the first length corresponding to the rectangular block when stacked and arranged in the first dimension is 64, then the contribution parameter corresponding to the rectangular block is 3-64=-61.

[0058] When the factor parameter for calculating the contribution parameter reference is one, for example, when the first length represents the contribution parameter, the second condition is that the first length is the smallest. That is, in some embodiments, the rectangular block to be arranged corresponding to the smallest first length is taken as the first rectangular block, and the target rectangular block is determined based on the first rectangular block.

[0059] It should be noted that, in the present application, any rectangular block in the first rectangular blocks can be determined as the target rectangular block, or a more suitable rectangular block can be selected therefrom. Further, in some embodiments, determining the target rectangular block based on the first rectangular block includes: when the first rectangular block is one, determining the first rectangular block as the target rectangular block.

[0060] It can be understood that during the arrangement process, it is inevitable that the contribution parameters corresponding to two or more rectangular blocks to be arranged are equal. In order to further obtain a target arrangement result with a smaller overall length, in some embodiments, the target rectangular block is determined based on the first rectangular block, and further includes: when there are two or more first rectangular blocks, the rectangular block to be arranged that does not correspond to the smallest first length is used as the second rectangular block, and after the first rectangular blocks are stacked and arranged in the first dimension, each of the second rectangular blocks is stacked and arranged in the first dimension to obtain the corresponding second arrangement results; the maximum second length of each of the second arrangement results corresponding to each of the first rectangular blocks in the first dimension (that is, the overall length of the second arrangement result in the first dimension) is determined; and the first rectangular block and the second rectangular block corresponding to the smallest second length are determined as the target rectangular block.

[0061] For example: From Table 1, we can see that the first rectangular block includes rectangular block No. 3, rectangular block No. 4 and rectangular block No. 6, and rectangular block No. 0 is a rectangular block in the currently determined arrangement result, then the second rectangular block includes rectangular block No. 1, rectangular block No. 2, rectangular block No. 5 and rectangular block No. 7. Based on the above method, the second arrangement results corresponding to rectangular block No. 3 are as follows: Figure 6As shown, they are 601 to 604, respectively. As shown in Table 2, the corresponding second lengths are 60 bytes, 52 bytes, 80 bytes, and 40 bytes, respectively. The second arrangement results corresponding to the No. 4 rectangular block are as follows Figure 6 As shown, they are 611 to 614 respectively, as shown in Table 2, and the corresponding second lengths are 60 bytes, 44 bytes, 72 bytes, and 40 bytes respectively; the second arrangement results corresponding to the No. 6 rectangular block are as follows Figure 6 As shown, they are 621 to 624 respectively. As shown in Table 2, the corresponding second lengths are 60 bytes, 36 bytes, 74 bytes, and 50 bytes respectively. It can be seen from Table 2 that among these second lengths, 36 bytes is the minimum value, so rectangular block No. 6 and rectangular block No. 2 corresponding to this value are determined as target rectangular blocks.

[0062] Table 2

[0063] Second arrangement results 0—>3—>1 0—>3—>2 0—>3—>5 0—>3—>7 Maximum second length 60 52 80 40 Second arrangement results 0—>4—>1 0—>4—>2 0—>4—>5 0—>4—>7 Maximum second length 60 44 72 40 Second arrangement results 0—>6—>1 0—>6—>2 0—>6—>5 0—>6—>7 Maximum second length 60 36 74 50

[0064] Step 405: Update the first arrangement result formed by stacking and arranging the target rectangular blocks in the first dimension and the currently determined arrangement result as the determined arrangement result.

[0065] by Figure 6 For example, the updated determined arrangement result is the arrangement result shown in 622 .

[0066] Step 406, determining whether there are the rectangular blocks to be arranged; if yes, returning to step 403; otherwise, executing step 407;

[0067] Step 407, taking the finally obtained determined arrangement result as the target arrangement result, and allocating corresponding memory to each of the computing nodes of the neural network based on the target arrangement result.

[0068] The present application embodiment further provides a memory allocation method, Figure 7 Schematic diagram of the implementation process of the memory allocation method of the embodiment of the present application, such as Figure 7 As shown, the method includes steps 701 to 706:

[0069] Step 701, determining the memory usage and memory occupation time information of each computing node of the neural network during operation;

[0070] Step 702: Based on the memory usage of the computing node and the time information of the memory occupation, define a rectangular block of the computing node; wherein the side length of the rectangular block in a first dimension represents the memory usage, the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension;

[0071] Step 703, enumerate the stacking arrangement of the rectangular blocks of each of the computing nodes of the neural network in the first dimension to obtain a third arrangement result.

[0072] In some embodiments, the stacking arrangement of the rectangular blocks of each of the computing nodes of the neural network in the first dimension can be exhaustively enumerated to obtain multiple third arrangement results.

[0073] Step 704: determine a maximum third length in the first dimension according to the third arrangement result.

[0074] It can be understood that the maximum third length is the entire length of the third arrangement result.

[0075] Step 705: Determine the arrangement result whose third length satisfies the first condition in the third arrangement result as the target arrangement result.

[0076] It should be noted that there is no restriction on the first condition and it can be set flexibly. For example, the first condition is that the overall length is the smallest; or the first condition is that the overall length is less than the length threshold. In short, as long as the memory overhead at the same time can be arranged to meet the memory overhead requirement.

[0077] For example, suppose the neural network includes computing nodes 0, 1, and 2, corresponding to rectangular blocks 0 to 2, respectively. Figure 8 As shown, the stacking arrangement of each rectangular block in the first dimension is exhaustively enumerated, and the third arrangement results obtained are 801-806 respectively. The maximum third lengths of each arrangement result are shown in Table 3, which are: 96 bytes, 64 bytes, 64 bytes, 64 bytes, 64 bytes and 96 bytes respectively. Therefore, the arrangement result corresponding to any minimum value 64 can be used as the target arrangement result, that is, "0—>2—>1", "1—>0—>2", "1—>2—>0", or "2—>0—>1" can be used as the target arrangement result.

[0078] Table 3

[0079] The third arrangement results 0—>1—>2 0—>2—>1 1—>0—>2 1—>2—>0 2—>0—>1 2—>1—>0 Maximum third length 96 64 64 64 64 96

[0080] Step 706: Allocate corresponding memory to each computing node of the neural network based on the target arrangement result.

[0081] In some embodiments, the memory allocation strategy can also adopt the first fit algorithm (First Fit), the best fit algorithm (Best Fit), the worst fit algorithm (Worst Fit) or the buddy algorithm (Buddy); among them, the basic principle of the first fit algorithm is: search the table from the first entry of the free partition table, and allocate the first free area that can meet the requirements to the job. The purpose of this method is to reduce the search time; the basic principle of the best fit algorithm is: find the free partition with the smallest size that can meet the job requirements from all free areas. This method can make the fragmentation as small as possible; the basic principle of the worst fit algorithm is: find the free partition with the largest size that can meet the job requirements from all free areas, so that the node size in the linked list tends to be uniform, which is suitable for systems with a narrow range of memory size requested for allocation; the basic principle of the buddy algorithm is: using the idea of ​​binary optimization, the memory is allocated in units of powers of 2, and only buddy memory blocks can be merged during merging.

[0082] In general, the basic idea of ​​these algorithms is to select a suitable continuous storage space based on the current memory usage and the future memory demand. However, since these memory allocation strategies are based on the unknown future memory demand, this is not completely consistent with the situation that the time sequence and size of memory demand are known after the neural network is fixed. Therefore, using these memory allocation strategies during the operation of the neural network will cause memory redundancy in time and space, wasting precious memory resources.

[0083] There are also relevant algorithms to optimize the characteristics of system memory allocation during the operation of neural networks.

[0084] Still Figure 2 Take the structure of the neural network shown in the figure as an example, Fig. 9 The time-memory size two-dimensional graph shown reflects this. Fig. 9 The horizontal axis in the middle represents the actual memory usage, that is, 0 corresponds to the starting position of the memory, and extending to the right represents the memory address gradually increasing in sequence; the vertical axis represents time, from top to bottom represents the corresponding memory distribution from the initial time as time increases. Each rectangular block represents the operator's overhead in the time dimension and memory usage dimension.

[0085] from Fig. 9 It can be seen that after computing nodes 2, 3, and 4 have finished running, computing node 5 can start using memory address 0 again, because when computing node 5 is doing calculations, the previous nodes 0 and 1 have already completed and can completely release the memory. Fig. 9By arranging the memory usage, we can get the minimum memory usage of the sum of the memory usage of computing nodes 1 to 5, that is, 152 storage units. Based on this representation, there are several algorithms to compress memory usage. However, these algorithms may be better than the greedy algorithm based on the total amount for some neural networks, but vice versa for some neural networks.

[0086] Based on this, an exemplary application of an embodiment of the present application in a practical application scenario will be described below.

[0087] In an embodiment of the present application, a memory allocation optimization scheme for neural network runtime based on the Russian algorithm is provided. The core idea of ​​this scheme is to search the memory allocation of the neural network runtime through a two-dimensional graph of time-memory size. The memory allocation process is converted into the optimal elimination process of Tetris to solve.

[0088] by Fig. 9 Take an example to illustrate the idea and process of conversion:

[0089] For accurate description, the memory requirement of each operator in the figure is modified to be more accurate, such as Fig.10 As shown in the left figure 1001. It can be seen that each rectangular block corresponds to an operator, wherein the width of the rectangular block on the horizontal axis represents the memory size required by the operator, and the length on the vertical axis represents the survival time of the memory block required by the operator.

[0090] Based on this, the specific steps of this solution are described as follows:

[0091] 1) If Fig.10 As shown in the left figure 1001, the known neural network is converted into a corresponding two-dimensional graph of time-memory size.

[0092] 2) Rotate the left image 1001 90 degrees counterclockwise around the center to obtain Fig.10 The schematic diagram shown in the right figure 1002.

[0093] 3) For the original problem, we need to find a solution that minimizes the memory overhead of a neural network. This can be understood from the time-memory diagram 1001 in the left figure as follows:

[0094] a) The position of each rectangular block in the figure is fixed on the vertical axis (time dimension) and not fixed on the horizontal axis (offset relative to address 0 in memory);

[0095] b) Seeking the solution with the minimum memory cost is to arrange the blocks on the horizontal axis while satisfying the vertical axis position of each rectangular block, so that after the final overall arrangement is determined, the projection distance on the horizontal axis (memory usage) is the shortest; in short, the final overall width is the smallest;

[0096] c) During the adjustment process in b), the rectangular blocks cannot overlap or overlap each other;

[0097] 4) Without making any modification to the left image 1001, the right image 1002 is obtained by simply rotating the image, and the corresponding problem in b) above is converted to: in the right image 1002, adjust the relative order of each rectangular block up and down to find the problem of the smallest overall height;

[0098] 5) Compared with the problem described in 4), we can first consider the solution in the Tetris game. Tetris is a game in which several blocks of different shapes are moved from top to bottom. When a row is complete, it will be eliminated, thus repeating the game over and over again. So when the order of appearance of the blocks is known, the problem that Tetris needs to solve is to adjust the position of the falling blocks so that the number of rows eliminated is the largest, or from another perspective, the overall height of all the blocks stacked together is the smallest.

[0099] 6) Comparing the descriptions of the two problems in 4) and 5), we can see that the solution to the problem in 4) can actually be converted into the solution to the problem in 5).

[0100] 7) Based on the conclusion of 6), take an algorithm as an example to demonstrate its implementation ideas. Here, we take the Pierre Dellacherie algorithm as an example. The strategy of this algorithm is: the lower the position of the plate is, the better, the closer the blocks are, the better, and the more blocks it contributes to the elimination of the row, the better. Based on this strategy, the algorithm divides each Tetris block into several units of small blocks, and uses 6 attributes to describe its specific attributes. It uses an evaluation function to calculate the score of each Tetris block placed in different positions, and returns the score of the optimal position as the result. Combined with the problem of solving the memory allocation of the neural network, it is necessary to add the information of the leftmost offset of the block placement in the evaluation function (because the left and right correspond to the information on the time dimension of the operator, which is fixed), and then adjust the order of appearance of each block to find a solution with the smallest height after the final placement, which is the optimal solution required.

[0101] For the idea of ​​this solution, the above steps are simply summarized as follows: Fig.11 As shown, as shown in 110, the original problem is to find the optimal memory allocation solution (that is, the rightmost boundary of the area composed of rectangular blocks is the smallest), as shown in 111, the new problem after conversion is to find the smallest upper boundary of the area composed of rectangular blocks. Based on this, as shown in 112, the same type of problem can be converted into a Tetris problem, that is, the original problem of finding the optimal solution for the insertion position (the least gap after the plate falls) is converted into finding the insertion position with the smallest overall upper limit.

[0102] According to this embodiment, the neural network memory allocation problem is converted, and the existing Tetris algorithm is used to find the optimal solution to the converted problem. On the basis of improving memory utilization, memory overhead is reduced, and the efficiency of neural network training and reasoning is improved.

[0103] Taking into account the characteristics of neural networks' runtime memory requirements, the relatively new problem of neural network memory allocation is converted into a relatively mature Tetris problem for solving, which improves memory utilization and reduces the overhead cost of neural network operation.

[0104] The technical solution provided in this application is applicable to various neural network frameworks, including but not limited to the current mainstream Caffe2, TensorFlow or PyTorch, and is also applicable to various hardware platforms.

[0105] In addition, the Tetris algorithm used in this application is only an example and can also be extended to other related and similar algorithms, including but not limited to algorithms designed for a specific hardware.

[0106] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0107] Based on the foregoing embodiments, an embodiment of the present application provides a memory allocation device, which includes the modules included and the units included in the modules, and can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0108] Fig.12 Schematic diagram of the structure of the memory allocation device according to the embodiment of the present application. Fig.12 As shown, the memory allocation device 12 includes:

[0109] A definition module 121 is configured to define a rectangular block of the computing node based on the memory usage and the time information of the memory usage of the computing node of the neural network; wherein the side length of the rectangular block in a first dimension represents the memory usage, the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension;

[0110] An arrangement module 122 is configured to stack and arrange the rectangular blocks of the computing nodes of the neural network in the first dimension, so as to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition;

[0111] The allocation module 123 is configured to allocate corresponding memory addresses to each of the computing nodes of the neural network based on the target arrangement result.

[0112] In some embodiments, the memory allocation device 12 further includes: a determination module configured to determine the memory usage and the time information of the memory occupied by each computing node of the neural network during operation before defining the rectangular block of the computing node.

[0113] In some embodiments, the arrangement module 122 is configured to: respectively evaluate the contribution parameters of each of the rectangular blocks to be arranged in a stacked arrangement in the first dimension to reducing the overall length of the target arrangement result; determine the target rectangular block based on the rectangular blocks to be arranged whose contribution parameters meet the second condition; update the first arrangement result composed of the target rectangular blocks when they are stacked in the first dimension and the currently determined arrangement result to the determined arrangement result; repeat the above steps until there are no rectangular blocks to be arranged, and use the determined arrangement result finally obtained as the target arrangement result.

[0114] In some embodiments, the arrangement module 122 is configured to: respectively determine the maximum first length of the first arrangement result formed by each of the rectangular blocks to be arranged and the currently determined arrangement result when they are stacked and arranged in the first dimension, and use the first length as the contribution parameter of the corresponding rectangular block; use the rectangular block to be arranged corresponding to the smallest first length as the first rectangular block, and determine the target rectangular block based on the first rectangular block.

[0115] In some embodiments, the arrangement module 122 is configured to: when the first rectangular block is one, determine the first rectangular block as a target rectangular block; when the first rectangular blocks are two or more, use the rectangular block to be arranged that is not corresponding to the smallest first length as a second rectangular block, and after the first rectangular blocks are stacked and arranged in the first dimension, stack and arrange each of the second rectangular blocks in the first dimension to obtain respective corresponding second arrangement results; determine the maximum second length of each of the second arrangement results corresponding to each of the first rectangular blocks in the first dimension; and determine the first rectangular block and the second rectangular block corresponding to the smallest second length as the target rectangular block.

[0116] In some embodiments, the arrangement module 122 is configured to: enumerate the stacking arrangement of the rectangular blocks of each of the computing nodes in the first dimension to obtain a third arrangement result; determine the maximum third length in the first dimension based on the third arrangement result; and determine the arrangement result in which the third length of the third arrangement result satisfies the first condition as the target arrangement result.

[0117] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0118] It should be noted that in the embodiments of this application Fig.12 The division of modules in the memory allocation device shown is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0119] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0120] An embodiment of the present application provides an electronic device, Fig.13 Schematic diagram of the hardware entity of the electronic device of the embodiment of the present application, such as Fig.13 As shown, the electronic device 130 includes a memory 131 and a processor 132, wherein the memory 131 stores a computer program that can be run on the processor 132, and the processor 132 implements the steps in the method provided in the above embodiment when executing the program.

[0121] It should be noted that the memory 131 is configured to store instructions and applications executable by the processor 132, and can also cache data to be processed or processed by the processor 132 and various modules in the electronic device 130 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0122] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0123] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0124] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0125] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0126] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0127] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0129] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0130] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0131] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0132] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0133] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0134] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0135] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0136] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A memory allocation method, It is characterized in that The method comprises: Based on the memory usage of the computing nodes of the neural network and the time information of the memory being occupied, a rectangular block of the computing nodes is defined; wherein the side length of the rectangular block in a first dimension represents the memory usage, the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension; The rectangular blocks of the computing nodes of the neural network are stacked and arranged in the first dimension, so as to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition; wherein the first condition includes one of the following: the overall length is less than a length threshold, and the overall length is the smallest; Based on the target arrangement result, assigning a corresponding memory address to each of the computing nodes of the neural network; The stacking and arranging of the rectangular blocks of the computing nodes of the neural network in the first dimension to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition includes: Evaluate respectively the contribution parameters of each rectangular block to be arranged in stacked arrangement in the first dimension to reducing the overall length of the target arrangement result; wherein the contribution parameters include: the maximum first length of the first arrangement result formed by the rectangular blocks to be arranged in stacked arrangement in the first dimension and the currently determined arrangement result, and / or the difference between the number of rectangular blocks contacted by the rectangular blocks to be arranged in stacked arrangement in the first dimension and the first length; Determine a target rectangular block based on the rectangular blocks to be arranged whose contribution parameters satisfy a second condition; wherein the second condition includes: the first length is minimum, and / or the difference is greater than a specific threshold; Updating a first arrangement result formed by stacking and arranging the target rectangular blocks in the first dimension and the currently determined arrangement result as the determined arrangement result; Repeat the above steps until there are no more rectangular blocks to be arranged, and use the finally obtained determined arrangement result as the target arrangement result; Alternatively, stacking and arranging the rectangular blocks of the computing nodes of the neural network in the first dimension to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition includes: List the stacking arrangement of the rectangular blocks of each of the computing nodes in the first dimension to obtain a third arrangement result; Determining a maximum third length in the first dimension according to the third arrangement result; The arrangement result in which the third length in the third arrangement result satisfies the first condition is determined as the target arrangement result.

2. The method according to claim 1, It is characterized in that Before defining the rectangular block of computing nodes, the method further includes: Determine the memory usage and the time information of the memory occupied by each computing node of the neural network during operation.

3. The method according to claim 1, It is characterized in that The respectively evaluating the contribution parameter of each rectangular block to be arranged in stacking arrangement in the first dimension to reducing the overall length of the target arrangement result includes: Determine respectively the maximum first length of a first arrangement result formed by stacking and arranging each of the rectangular blocks to be arranged in the first dimension with the currently determined arrangement result, and use the first length as a contribution parameter of the corresponding rectangular block; Accordingly, determining the target rectangular block based on the rectangular blocks to be arranged whose contribution parameters satisfy the second condition includes: The rectangular block to be arranged corresponding to the smallest first length is taken as a first rectangular block, and the target rectangular block is determined based on the first rectangular block.

4. The method according to claim 3, It is characterized in that The step of determining the target rectangular block based on the first rectangular block includes: When there is one first rectangular block, the first rectangular block is determined as the target rectangular block.

5. The method according to claim 4, It is characterized in that Also includes: When there are two or more first rectangular blocks, the rectangular blocks to be arranged that are not corresponding to the smallest first length are used as second rectangular blocks, and after the first rectangular blocks are stacked and arranged in the first dimension, the second rectangular blocks are stacked and arranged in the first dimension to obtain respective corresponding second arrangement results; Determine the maximum second length of each of the second arrangement results corresponding to each of the first rectangular blocks in the first dimension; The first rectangular block and the second rectangular block corresponding to the smallest second length are determined as the target rectangular block.

6. A memory allocation device, It is characterized in that include: A definition module is configured to define a rectangular block of the computing node based on the memory usage and the time information of the memory occupation of the computing node of the neural network; wherein the side length of the rectangular block in a first dimension represents the memory usage, the side length of the rectangular block in a second dimension represents the time information, and the first dimension is perpendicular to the second dimension; An arrangement module is configured to stack and arrange the rectangular blocks of each computing node of the neural network in the first dimension, so as to arrange a target arrangement result whose overall length in the first dimension satisfies a first condition; wherein the first condition includes one of the following: the overall length is less than a length threshold, and the overall length is minimum; an allocation module, configured to allocate a corresponding memory address to each of the computing nodes of the neural network based on the target arrangement result; The arrangement module is configured to respectively evaluate the contribution parameters of each rectangular block to be arranged in stacking arrangement on the first dimension to reduce the overall length of the target arrangement result; wherein the contribution parameters include: the maximum first length of the first arrangement result formed by the rectangular blocks to be arranged in stacking arrangement on the first dimension and the currently determined arrangement result, and / or the difference between the number of rectangular blocks contacted by the rectangular blocks to be arranged in stacking arrangement on the first dimension and the first length; determine the target rectangular block based on the rectangular blocks to be arranged that meet the second condition according to the contribution parameters; wherein the second condition includes: the first length is the smallest, and / or the difference is greater than a specific threshold; update the first arrangement result formed by the target rectangular block in stacking arrangement on the first dimension and the currently determined arrangement result to the determined arrangement result; repeat the above steps until there are no rectangular blocks to be arranged, and use the finally obtained determined arrangement result as the target arrangement result; The arrangement module is configured to enumerate the stacking arrangement of the rectangular blocks of each of the computing nodes in the first dimension to obtain a third arrangement result; determine the maximum third length in the first dimension based on the third arrangement result; and determine the arrangement result in which the third length satisfies the first condition in the third arrangement result as the target arrangement result.

7. The device according to claim 6, It is characterized in that The device also includes: The determination module is configured to determine the memory usage and the time information of the memory occupied by each computing node of the neural network during operation before defining the rectangular block of the computing nodes.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images