A method and apparatus for optimizing a neural network model

Abstract

This application discloses a method and apparatus for optimizing a neural network model, relating to the field of communication technology. It eliminates the need for model retraining, reducing time consumption. The specific scheme is as follows: the input data of the neural network model is divided into M data blocks; M is greater than or equal to 2, and M is an integer; the M data blocks are sequentially calculated layer by layer, from the 1st to the Kth network layer; wherein the weight corresponding to the i-th network layer of each data block is the same as the weight corresponding to the i-th network layer in the neural network model, i is greater than or equal to 1 and less than or equal to K, and i is an integer; the calculation results of the M data blocks at the Kth network layer are combined and used as the input to the (K+1)-th network layer.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method and apparatus for optimizing a neural network model. Background Technology

[0002] Currently, convolutional neural networks (CNNs) are widely used in image recognition. During the inference phase, as the feature map resolution decreases, the memory occupied by the computation results of each network layer exhibits an inverted cone-shaped structure. In embedded systems with limited memory, this uneven memory distribution leads to significant memory waste, hindering the deployment of deep learning in memory-constrained embedded systems.

[0003] Existing technologies reduce memory usage by employing techniques such as channel dimensionality reduction and low-bit quantization. However, these techniques do not fundamentally change the "top-heavy" nature of network memory usage. Moreover, these solutions alter the network structure, requiring network retraining, which is time-consuming. Summary of the Invention

[0004] This application provides a method and apparatus for optimizing a neural network model, which eliminates the need to retrain the model and reduces time consumption.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] A first aspect of this application provides a method for optimizing a neural network model, the neural network model comprising N network layers, where N is greater than or equal to 2 and N is an integer. The method comprises: dividing the input data of the neural network model into M data blocks, where M is greater than or equal to 2 and M is an integer; sequentially calculating the M data blocks layer by layer from the 1st network layer to the Kth network layer, where K is greater than or equal to 1 and less than N and K is an integer; wherein the weight corresponding to the i-th network layer of each data block is the same as the weight corresponding to the i-th network layer in the neural network model, where i is greater than or equal to 1 and less than or equal to K and i is an integer; and merging the calculation results of the M data blocks at the Kth network layer as the input of the (K+1)-th network layer.

[0007] Based on this scheme, the input data is divided into M data blocks. Each data block shares the weights of the neural network model in the corresponding network layers from the 1st to the Kth, thus eliminating the need to retrain the model and saving time. Furthermore, by dividing the input data of the neural network model and sequentially calculating the M data blocks from the 1st to the Kth network layers, and then merging the calculation results of the Kth network layer at the (K+1)th network layer, the memory among the M data blocks can be reused in subsequent memory reuse processes, reducing memory consumption and ensuring that the effect of splitting and merging is the same as that without splitting.

[0008] In conjunction with the first aspect, in one possible implementation, the dimensions of the input data include width w, height h, and depth c. The above-mentioned division of the input data of the neural network model into M data blocks includes: dividing the input data of the neural network model into M data blocks along the width w dimension.

[0009] Based on this scheme, by segmenting the input data along the width dimension, compared to segmenting along the depth dimension, the correlation between features can be guaranteed, ensuring that the information richness of each neuron is not reduced, thus ensuring that the effect and accuracy are not affected. Moreover, when merging the calculation results after calculating the Kth network layer, the arrangement of the merged data is the same as that of the unsegmented data.

[0010] In combination with the first aspect and the above possible implementations, in another possible implementation, the above-mentioned method of dividing the input data of the neural network model into M data blocks along the width w dimension includes: uniformly dividing the input data of the neural network model into M data blocks along the width w dimension; or, non-uniformly dividing the input data of the neural network model into M data blocks along the width w dimension.

[0011] Based on this scheme, when dividing the input data into M data blocks, either uniform or non-uniform partitioning can be used. Understandably, when dividing the input data of a neural network model into M data blocks along the width w dimension, uniform partitioning is simpler to implement than non-uniform partitioning, and uniform partitioning also performs better in terms of memory reuse than non-uniform partitioning.

[0012] Combining the first aspect and the above possible implementations, in another possible implementation, the calculation result of the Pth data block at the Jth layer reuses the memory occupied by the useless data in the historical calculation results, where P is any positive integer greater than or equal to 1 and less than or equal to M, and J is any positive integer greater than 1 and less than or equal to K.

[0013] Based on this scheme, by reusing the memory occupied by useless data in historical calculation results, memory consumption can be greatly reduced, allowing neural networks to be deployed in embedded systems with limited memory. It can be understood that "historical calculation results" refers to the data and calculation results stored before the current network layer (e.g., the Pth data block is in the Jth network layer) under the premise of memory reuse. "Useless data" in historical calculation results refers to data stored before the current network layer that will not be used when calculating the calculation results of the current network layer and subsequent network layers, under the premise of memory reuse.

[0014] In conjunction with the first aspect and the above possible implementations, in another possible implementation, the M data blocks include a first data block and a second data block, the K network layers include a first network layer and a second network layer, the calculation result of the first data block in the first network layer is a first calculation result, the calculation result of the first data block in the second network layer is a second calculation result, the calculation result of the second data block in the first network layer is a third calculation result, the calculation result of the second data block in the second network layer is a fourth calculation result, the starting address of the first data block is a first offset position, the starting address of the second data block is a second offset position, and the starting address of the first calculation result is a third offset position; if the second calculation result is obtained, the first data block is useless data, and the second calculation result reuses the memory occupied by the first data block.

[0015] Based on this scheme, since the first data block is useless data when the second calculation result is obtained, that is, the data of the first data block will not be used when calculating the calculation results of the current network layer and the network layers after the current network layer, the second calculation result can reuse the memory occupied by the first data block to reduce memory consumption.

[0016] In conjunction with the first aspect and the above possible implementations, in another possible implementation, the second calculation result reuses the memory occupied by the first data block, specifically including: if the second calculation result is less than or equal to the memory occupied by the first data block, storing the second calculation result in memory with the first offset position as the starting address; if the second calculation result is greater than the memory occupied by the first data block, releasing the memory occupied by the first data block, and shifting the storage positions of the second data block and the first calculation result forward, so that the starting address of the second data block is the first offset position, and storing the second calculation result continuously after the first calculation result; or, if the second calculation result is greater than the memory occupied by the first data block, storing a portion of the second calculation result in the memory occupied by the first data block, and storing the remaining portion of the second calculation result continuously after the first calculation result.

[0017] Based on this scheme, when reusing memory occupied by useless data, reuse can be based on the relationship between the size of the useless data and the size of the calculation result of the current network layer. For example, when the calculation result of the current network layer is larger than the memory occupied by the useless data, the calculation result of the current network layer can be stored continuously in memory or stored in segments. It is understandable that storing the calculation result of the current network layer continuously in memory only requires recording an offset position and size. Compared to storing the calculation result of the current network layer in segments, less information is recorded, thus saving more memory.

[0018] In combination with the first aspect and the above possible implementations, in another possible implementation, the method further includes: if the first calculation result is useless data when the third calculation result is obtained, the third calculation result reuses the memory occupied by the first calculation result; the fourth calculation result reuses the free memory between storing the second calculation result and storing the third calculation result.

[0019] Based on this scheme, since the first calculation result is useless data when the third calculation result is obtained, that is, the first calculation result will not be used when calculating the calculation results of the current network layer (the second data block is in the first network layer) and the network layers after the current network layer, the third calculation result can reuse the memory occupied by the first calculation result to reduce memory consumption.

[0020] In combination with the first aspect and the above possible implementations, in another possible implementation, the method further includes: determining the merging layer as the Kth network layer; wherein, under the premise of memory reuse, the memory occupied by storing the calculation results of the Kth network layer and the data before the calculation results of the Kth network layer is the peak memory max_tmpbuf, and the peak memory is the maximum memory that can be occupied by all existing calculation results of the neural network model; the calculation results of the (K+1)th to Nth network layers reuse the peak memory.

[0021] Based on this scheme, under the premise of memory reuse, the network layer corresponding to the peak memory can be determined as the merging layer K. This allows the calculation results of the network layers after merging layer K to reuse the peak memory, thereby ensuring that the hardware's computing performance is not greatly affected and reducing memory consumption.

[0022] Combining the first aspect and the above possible implementations, in another possible implementation, the method further includes: calculating the sum of memory occupied by the calculation results from the (K+1)th network layer to the Nth network layer (addbuf); The number of blocks to be divided is determined to be M; where, This indicates rounding down from max_tmpbuf / addbuf.

[0023] Based on this scheme, under the premise of memory reuse, the number of blocks to be divided can be determined based on the peak memory and the memory occupied by the calculation results after the merging layer.

[0024] In combination with the first aspect and the above possible implementations, in another possible implementation, the method further includes: determining a first average precision mAP based on the number of data blocks M and the merging layer; wherein, the weight corresponding to each data block in the nth network layer is the same as the weight corresponding to the nth network layer in the above neural network model, n is greater than or equal to 1 and less than or equal to K, and n is an integer; if the difference between the first average precision and the second average precision mAP when the input data is not segmented is greater than a preset threshold, a preset algorithm is used to update the number of data blocks M to M' and the merging layer K to K', where M' is less than M and K' is less than K.

[0025] Based on this scheme, the mAP before and after segmentation can be compared. If the mAP after segmentation differs significantly from the mAP before segmentation, segmentation can be performed at a shallower layer, that is, the merged layer is updated to a shallower network layer to ensure that the mAP after segmentation is basically consistent with the mAP before segmentation, thereby ensuring that the accuracy of the segmented model is basically unaffected.

[0026] A second aspect of this application provides an optimization apparatus for a neural network model, the neural network model including N network layers, where N is greater than or equal to 2 and N is an integer. The apparatus includes: a segmentation unit for segmenting the input data of the neural network model into M data blocks, where M is greater than or equal to 2 and M is an integer; a processing unit for sequentially calculating the M data blocks layer by layer from the 1st network layer to the Kth network layer, where K is greater than or equal to 1 and less than N and K is an integer; wherein the weight corresponding to the i-th network layer of each data block is the same as the weight corresponding to the i-th network layer in the neural network model, i is greater than or equal to 1 and less than or equal to K and i is an integer; and a merging unit for merging the calculation results of the M data blocks in the Kth network layer and using them as the input of the (K+1)-th network layer.

[0027] In conjunction with the second aspect, in one possible implementation, the dimensions of the input data include width w, height h, and depth c; the segmentation unit is specifically used to segment the input data of the neural network model into M data blocks along the width w dimension.

[0028] In combination with the second aspect and the above possible implementations, in another possible implementation, the above-mentioned segmentation unit is specifically used to uniformly segment the input data of the above neural network model into M data blocks in the width w dimension; or, to non-uniformly segment the input data of the above neural network model into M data blocks in the width w dimension.

[0029] Combining the second aspect and the above possible implementations, in another possible implementation, the calculation result of the Pth data block at the Jth layer reuses the memory occupied by the useless data in the historical calculation results, where P is any positive integer greater than or equal to 1 and less than or equal to M, and J is any positive integer greater than 1 and less than or equal to K.

[0030] In conjunction with the second aspect and the above possible implementations, in another possible implementation, the M data blocks include a first data block and a second data block, the K network layers include a first network layer and a second network layer, the calculation result of the first data block in the first network layer is a first calculation result, the calculation result of the first data block in the second network layer is a second calculation result, the calculation result of the second data block in the first network layer is a third calculation result, the calculation result of the second data block in the second network layer is a fourth calculation result, the starting address of the first data block is a first offset position, the starting address of the second data block is a second offset position, and the starting address of the first calculation result is a third offset position; the processing unit is specifically used to determine that if the processing unit determines that the second calculation result is obtained, the first data block is useless data, and the second calculation result reuses the memory occupied by the first data block.

[0031] In conjunction with the second aspect and the above possible implementations, in another possible implementation, the processing unit is specifically configured to: when the second calculation result is less than or equal to the memory occupied by the first data block, store the second calculation result in memory with the first offset position as the starting address; when the second calculation result is greater than the memory occupied by the first data block, release the memory occupied by the first data block, and move the storage positions of the second data block and the first calculation result forward, such that the starting address of the second data block is the first offset position, and continuously store the second calculation result after the first calculation result; or, when the second calculation result is greater than the memory occupied by the first data block, store a portion of the second calculation result in the memory occupied by the first data block, and continuously store the remaining portion of the second calculation result after the first calculation result.

[0032] In conjunction with the second aspect and the above possible implementations, in another possible implementation, the processing unit is further configured to: if the processing unit determines that the first calculation result is useless data when the third calculation result is obtained, the third calculation result reuses the memory occupied by the first calculation result; the fourth calculation result reuses the free memory between storing the second calculation result and storing the third calculation result.

[0033] In conjunction with the second aspect and the above possible implementations, in another possible implementation, the processing unit is further configured to: determine the merging layer as the Kth network layer; wherein, under the premise of memory reuse, the memory occupied by storing the calculation results of the Kth network layer and the data before the calculation results of the Kth network layer is the peak memory max_tmpbuf, and the peak memory is the maximum memory that can be occupied by all existing calculation results of the neural network model; the calculation results of the (K+1)th to Nth network layers reuse the peak memory.

[0034] In conjunction with the second aspect and the above possible implementations, in another possible implementation, the processing unit is further configured to: calculate the sum of memory occupied by the calculation results from the (K+1)th network layer to the Nth network layer (addbuf); and... The number of blocks to be divided is determined to be M; where, This indicates rounding down from max_tmpbuf / addbuf.

[0035] In conjunction with the second aspect and the above possible implementations, in another possible implementation, the processing unit is further configured to: determine a first average precision mean mAP based on the number of data blocks M and the merging layer; wherein, the weight corresponding to the nth network layer of each of the M data blocks is the same as the weight corresponding to the nth network layer in the above neural network model, n is greater than or equal to 1 and less than or equal to K, and n is an integer; if the difference between the first average precision mean and the second average precision mean mAP when the input data is not segmented is greater than a preset threshold, a preset algorithm is used to update the number of data blocks M to M' and the merging layer K to K', where M' is less than M and K' is less than K.

[0036] The effects of the second aspect and its various implementations can be described by referring to the effects of the first aspect and its various implementations, and will not be repeated here.

[0037] A third aspect of this application provides a computer storage medium storing computer program code, which, when run on a computer or processor, causes the computer or processor to execute the optimization method of the aforementioned neural network model.

[0038] A fourth aspect of this application provides a computer program product that stores computer software instructions executed by the processor, the computer software instructions including an optimization method for executing the neural network model.

[0039] A fifth aspect of this application provides an optimization apparatus for a neural network model. The apparatus exists in the form of a chip. The structure of the apparatus includes a processor and optionally a memory. The memory is coupled to the processor to store necessary program instructions and data of the apparatus. The processor is used to execute the program instructions stored in the memory, so that the apparatus performs the above-described optimization method for the neural network model. Attached Figure Description

[0040] Figure 1 A schematic diagram illustrating the memory consumption of a neural network provided in an embodiment of this application;

[0041] Figure 2 A schematic diagram illustrating an application scenario of a neural network model optimization method provided in this application embodiment;

[0042] Figure 3 A flowchart illustrating a method for optimizing a neural network model provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of a neural network model provided in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of another neural network model provided in an embodiment of this application;

[0045] Figure 6 A schematic diagram of the memory distribution for an optimization method of a neural network model provided in this application embodiment. Figure 1 ;

[0046] Figure 7a A schematic diagram of the memory distribution for an optimization method of a neural network model provided in this application embodiment. Figure 2 ;

[0047] Figure 7b A schematic diagram of the memory distribution for an optimization method of a neural network model provided in this application embodiment. Figure 3 ;

[0048] Figure 7c A schematic diagram of the memory distribution for an optimization method of a neural network model provided in this application embodiment. Figure 4 ;

[0049] Figure 7d A schematic diagram of the memory distribution for an optimization method of a neural network model provided in this application embodiment. Figure 5 ;

[0050] Figure 7e A schematic diagram of the memory distribution for an optimization method of a neural network model provided in this application embodiment. Figure 6 ;

[0051] Figure 8 This is a schematic diagram illustrating the arrangement of merged positional data in an optimization method for a neural network model provided in an embodiment of this application.

[0052] Figure 9 A flowchart illustrating another method for optimizing a neural network model provided in an embodiment of this application;

[0053] Figure 10 A flowchart illustrating another method for optimizing a neural network model provided in an embodiment of this application;

[0054] Figure 11 A flowchart illustrating another method for optimizing a neural network model provided in an embodiment of this application;

[0055] Figure 12 A schematic diagram of the PR of an unsegmented scheme and a segmented scheme for mAP provided in an embodiment of this application;

[0056] Figure 13a A schematic diagram illustrating the memory usage of an unsegmented scheme provided in an embodiment of this application;

[0057] Figure 13b A schematic diagram illustrating the memory usage of a segmentation scheme provided in an embodiment of this application;

[0058] Figure 14 A schematic diagram of the structure of an optimization device for a neural network model provided in an embodiment of this application;

[0059] Figure 15 This is a schematic diagram of the structure of an optimization device for a neural network model provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural.

[0061] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0062] For example, during the inference phase of a convolutional neural network, as the feature map resolution decreases, the memory occupied by the computation results of each network layer exhibits an inverted cone-shaped structure. For instance, as... Figure 1 The convolutional neural network shown occupies 70% of the total network memory in convolutional layers 1 (conv1) and 2 (conv2). Subsequent layers, after multiple pooling operations, experience a decrease in feature map resolution and consume significantly less memory than conv1 and conv2. Therefore, in embedded systems with limited memory, this unbalanced memory distribution in an inverted cone structure leads to substantial memory waste, hindering the deployment of deep learning in memory-constrained embedded systems.

[0063] This application provides a method for optimizing a neural network model. This method eliminates the need to retrain the model and reduces memory consumption, enabling the neural network to be deployed in embedded systems with limited memory.

[0064] For example, the neural network model optimization method provided in this application can be applied to camera security scenarios as well as other image recognition scenarios, and this application does not limit its application. The following embodiments are only illustrated using an application to camera security scenarios.

[0065] Figure 2 This is a schematic diagram illustrating an application scenario of a neural network model optimization method provided in an embodiment of this application. For example... Figure 2 As shown, the video input (VI) device captures raw images, which are then processed by the video processing subsystem (VPSS) to scale the video frames to the network's input resolution. The scaled images are then loaded into different offline models via the intelligent video process (IVP) module to manage tasks such as face detection (FD), human detection (PD), and face recognition (FR). These different offline models can perform single-object detection (e.g., detecting only faces or only human shapes) or multi-object detection (e.g., simultaneously detecting faces, human shapes, and vehicles). Figure 2 The Smart Vision Platform-algorithm (SVP_ALGO) integrates algorithm modules corresponding to different offline models. The SVP_ALGO algorithm module includes a Video Graphics Sub-System (VGS) and a Motion Detection (MD) module. The MD module is an algorithm module encapsulated using operators from the Intelligen Video Engine (IVE) module. VGS performs color space conversion on video frames processed by VPSS and performs mean subtraction and variance division on the video frames. VGS can also scale the region of interest (ROI) in the image. The Intelligen Video Engine (IVE) module can manage different models, forming a model chain from the various network layers in the model. The processed video frame data is then fed into the model chain and then sent to hardware (logic) processing to obtain the final result. The Video Encoding (VENC) module is used to encode the detected ROI.

[0066] The neural network model optimization method provided in this application embodiment is applied in... Figure 2 In the scenario shown, the neural network model can be optimized to reduce the memory consumption of the embedded system when the neural network model runs on a system with limited memory.

[0067] Combination Figure 1 and Figure 2 ,like Figure 3 As shown, this application provides an embodiment of a method for optimizing a neural network model. The neural network model includes N network layers, where N is greater than or equal to 2 and N is an integer. The method includes steps S301-S303.

[0068] S301. Divide the input data of the neural network model into M data blocks.

[0069] Where M is greater than or equal to 2, and M is an integer. For example, the dimensions of the input data include width w, height h, and depth c. In step S301, the input data can be divided into M data blocks along the width w, height h, or depth c dimension. This embodiment of the application does not limit this. The following embodiment only illustrates the example of dividing the input data into M data blocks along the width w dimension.

[0070] For example, the neural network model described above may include N network layers and parameters such as weights corresponding to each network layer. This neural network model can be a model trained with Caffe, i.e., a caffemodel. Optionally, this neural network model can also be a model trained with other frameworks such as TensorFlow or PyTorch, and then converted into a caffemodel.

[0071] For example, such as Figure 4 As shown, the neural network model can include five network layers. The first network layer is a convolutional layer, denoted as conv1; the second network layer is a convolutional layer, denoted as conv2; the third network layer is a pooling layer, denoted as pooling layer 1; the fourth network layer is a pooling layer, denoted as pooling layer 2; and the fifth network layer is a fully connected layer. The weights corresponding to the first network layer conv1 are W1, the weights corresponding to the second network layer conv2 are W2, and the weights corresponding to the fifth network layer (fully connected layer) are W3. This application does not limit the specific structure of the neural network model, but only uses [specific examples]. Figure 4 The neural network structure shown is used as an example for illustration.

[0072] For example, the dimensions of the input data mentioned above include width w, height h, and depth c. Optionally, the dimensions of the input data may also include n dimensions, where n represents the number of batches, i.e., the number of feature maps or input images. This application does not limit the value of n in its embodiments; the following embodiments only illustrate an example with n as 1.

[0073] For example, in one implementation, dividing the input data into M data blocks along the width w dimension can be achieved by uniformly dividing the input data into M data blocks along the width dimension. For instance, taking an input data dimension of 6*6*3 and M as 2, step S301 can divide the 6*6*3 input data into two data blocks along the w dimension, with each data block having a dimension of 3*6*3.

[0074] For example, consider dividing the input data into two blocks. Figure 5 As shown, the input data w*h*c can be divided into two data blocks along the w dimension, where the first data block has a dimension of w / 2*h*c and the second data block has a dimension of w / 2*h*c.

[0075] For example, when uniformly dividing the input data into M data blocks along the width dimension w, M can be divided by the dimension of the input data along the w dimension, that is, the dimension of the input data along the w dimension is an integer multiple of the number of blocks M, thus dividing the input data into M data blocks along the w dimension. Optionally, when the number of blocks M is not divisible by the dimension of the input data along the w dimension, the value of the number of blocks M can be updated to the largest integer that is divisible by the dimension of the input data along the w dimension and is less than M. For example, when the dimension of the input data along the w dimension is 6 and the number of blocks M is 4, since it is impossible to uniformly divide the input data into 4 data blocks along the w dimension, the value of the number of blocks M can be determined to be the largest integer that is divisible by the dimension 6 and is less than 4, that is, M can be updated to 3. That is, the input data can be uniformly divided into 3 data blocks along the w dimension.

[0076] For example, in another implementation, the above-mentioned division of the input data into M data blocks along the width w dimension can be achieved by unevenly dividing the input data into M data blocks along the width dimension. For instance, taking the input data dimension as 6*6*3 and M as 2, step S301 can divide the 6*6*3 input data into two data blocks along the w dimension, one data block having a dimension of 4*6*3 and the other data block having a dimension of 2*6*3.

[0077] It should be noted that the embodiments of this application do not limit the specific implementation method of dividing the input data of the neural network model into M data blocks along the width w dimension. For example, a uniform division method or a non-uniform division method can be used. It should be noted that when dividing the input data of the neural network model into M data blocks along the width w dimension, the uniform division method is simpler to implement than the non-uniform division method.

[0078] S302. Calculate the M data blocks one by one from the first network layer to the Kth network layer.

[0079] Where K is greater than or equal to 1 and less than N, and K is an integer. This Kth network layer can be called a merging layer.

[0080] The weight corresponding to the i-th network layer of each data block is the same as the weight corresponding to the i-th network layer in the neural network model, where i is greater than or equal to 1 and less than or equal to K, and i is an integer. That is, the weight corresponding to each data block in the M data blocks in the 1st to Kth network layers is the same as the weight corresponding to the corresponding network layer in the neural network model. In other words, each data block can share the weights corresponding to the neural network model. It can be understood that, in this embodiment of the application, before the merging layer is merged, each data block in the M data blocks can share the weights of the corresponding network layers in the neural network model in the 1st to Kth network layers.

[0081] For example, taking the Kth network layer as the second network layer, that is, merging layers become the second network layer. Figure 5 As shown, the first and second data blocks are calculated layer by layer from the first to the second network layer. The weight corresponding to the first data block in the first network layer conv1-1 is... Figure 4 The neural network model shown has weight W1 corresponding to the first network layer conv1; the weights corresponding to the second data block in the first network layer conv1-2 are... Figure 4 The neural network model shown has weight W1 corresponding to the first network layer conv1; the weight W1 corresponding to the first data block in the second network layer conv2-1 is... Figure 4 The neural network model shown has weight W2 corresponding to the second network layer conv2; the weight of the second data block corresponding to the second network layer conv2-2 is... Figure 4 The neural network model shown has weight W2 corresponding to the second network layer conv2. That is, the weights of each data block in each network layer can share the weights of the corresponding network layer in the neural network model.

[0082] Understandably, since the weights of the segmented data blocks in this embodiment can share the weights of the neural network model in the corresponding network layers after the input data is segmented, there is no need to retrain the neural network model. Compared with the prior art, which requires retraining the network after changing the network structure, this is more time-saving.

[0083] For example, in step S302 above, sequentially calculating the M data blocks from the first network layer to the Kth network layer can include: performing calculations sequentially from the first network layer to the Kth network layer among the M data blocks. For example, combined with Figure 5 As shown, the calculation results of the first data block in the first network layer conv1-1, the first data block in the second network layer conv2-1, the second data block in the first network layer conv1-2, and the second data block in the second network layer conv2-2 are calculated sequentially. It can be understood that the calculation result of the first data block in the first network layer refers to the result after the first data block has undergone calculation in the first network layer. The calculation result of the first data block in the second network layer refers to the result after the first data block has undergone calculation in both the first and second network layers. That is, the calculation result of the Mth data block in the Kth network layer refers to the result after the Mth data block has undergone calculation in all three network layers (from the first to the Kth).

[0084] For example, after calculating the M data blocks layer by layer from the first network layer to the Kth network layer, the calculation results can be stored in memory sequentially.

[0085] For example, the calculation result of the Pth data block in the above M data blocks at the Jth level can reuse the memory occupied by useless data in the historical calculation results, where P is any positive integer greater than or equal to 1 and less than or equal to M, and J is any positive integer greater than 1 and less than or equal to K.

[0086] The aforementioned historical calculation results refer to the data and calculation results stored before the current network layer (e.g., the Pth data block is in the Jth network layer) under the premise of memory reuse. Useless data in historical calculation results refers to data stored before the current network layer that will not be used when calculating the calculation results of the current network layer and subsequent network layers, under the premise of memory reuse.

[0087] For example, such as Figure 5As shown, taking the current network layer as conv1-1 as an example, the calculation result of the Pth data block in layer J is the calculation result of the 1st data block in the 1st network layer. Since there is no memory reuse before conv1-1, the historical calculation result refers to the data stored before conv1-1, including the data of the 1st data block and the data of the 2nd data block. This historical calculation result, including the data of the 1st and 2nd data blocks, will be used when calculating the calculation results of conv1-1 and conv1-2. Therefore, when the current network layer is conv1-1, there is no useless data in the historical calculation result. Thus, the calculation result of the 1st data block in the 1st network layer has no memory that can be reused.

[0088] For example, taking the current network layer as conv2-1, the calculation result of the Pth data block in layer J is the calculation result of the first data block in the second network layer. Since there is no memory reuse before conv2-1, the historical calculation result refers to the data and calculation results stored before conv2-1, including the data of the first data block, the data of the second data block, and the calculation result of conv1-1. In this historical calculation result, the data of the first data block will not be used when calculating the calculation results of conv2-1 and subsequent network layers. Therefore, when the current network layer is conv2-1, the data of the first data block in the historical calculation result is useless data. Therefore, the calculation result of the first data block in the second network layer can reuse the useless data in the historical calculation result, that is, the calculation result of the first data block in the second network layer (conv2-1) can reuse the memory occupied by the data of the first data block. Understandably, since different data blocks are calculated sequentially from the first network layer to the Kth network layer, the network layers after conv2-1 include conv1-2, conv2-2, pooling layer 1, pooling layer 2, and fully connected layer.

[0089] For example, consider a network layer of conv1-2 where the calculation result of conv2-1 is the same size as the first data block. The calculation result of the Pth data block at layer J is the calculation result of the second data block at the first network layer. Since the calculation result of conv2-1 reuses the memory occupied by the data of the first data block, under the premise of memory reuse, the historical calculation result is the data and calculation result stored before conv1-2. This historical calculation result includes the calculation result of conv1-1, the calculation result of conv2-1, and the data of the second data block. In this historical calculation result, the calculation result of conv1-1 will not be used when calculating the calculation results of conv1-2 and subsequent network layers. Therefore, when the current network layer is conv1-2, the useless data in the historical calculation result is the calculation result of conv1-1. Therefore, the computation result of the second data block in the first network layer (conv1-2) can reuse useless data from the historical computation results; that is, the computation result of the first data block in the second network layer (conv1-2) can reuse the memory occupied by the computation result of conv1-1. The network layers after conv1-2 include conv2-2, pooling layer 1, pooling layer 2, and fully connected layers.

[0090] For example, consider M data blocks including a first data block and a second data block, and K network layers including a first network layer and a second network layer. The calculation result of the first data block in the first network layer is a first calculation result; the calculation result of the first data block in the second network layer is a second calculation result; the calculation result of the second data block in the first network layer is a third calculation result; and the calculation result of the second data block in the second network layer is a fourth calculation result. The starting address of the first data block is a first offset position, the starting address of the second data block is a second offset position, and the starting address of the first calculation result is a third offset position. If the second calculation result is obtained, and the first data block is useless data, the second calculation result reuses the memory occupied by the first data block. It is understood that the above-mentioned calculation result of the first data block in the second network layer refers to the result of the first data block after calculation by the first network layer and the second network layer. The above-mentioned calculation result of the second data block in the second network layer refers to the result of the second data block after calculation by the first network layer and the second network layer. The calculation result of the first data block in the second network layer is the result of the first calculation result after being calculated by the second network layer, and the calculation result of the second data block in the second network layer is the result of the third calculation result after being calculated by the second network layer.

[0091] For example, combining Figure 5 ,like Figure 6As shown, taking M data blocks (including the first and second data blocks) and K network layers (including the first and second network layers) as an example, the starting address of the first data block is stored at offset 1, and the starting address of the second data block is stored at offset 2. The calculation result of the first data block in the first network layer (first calculation result) is denoted as conv1-1, and the starting address of conv1-1 is stored at offset 3. The calculation result of the first data block in the second network layer (second calculation result) is denoted as conv2-1, the calculation result of the second data block in the first network layer (third calculation result) is denoted as conv1-2, and the calculation result of the second data block in the second network layer (fourth calculation result) is denoted as conv2-2. Figure 5 In the neural network model shown, when calculating the result conv2-1 of the first data block in the second network layer, the first data block is useless data, and the result conv2-1 of the first data block in the second network layer can reuse the memory occupied by the first data block.

[0092] For example, the above-mentioned second calculation result reuses the memory occupied by the first data block, specifically including: when the second calculation result is less than or equal to the memory occupied by the first data block, storing the second calculation result in memory with the first offset position as the starting address.

[0093] For example, combining Figure 5 ,like Figure 7a As shown, conv2-1 (the second calculation result) is less than the memory occupied by the first data block (useless data), so conv2-1 can be stored in the memory starting at the first offset position. That is, conv2-1 can reuse the memory occupied by the first data block.

[0094] For example, the above-mentioned reuse of the memory occupied by the first data block for the second calculation result specifically includes: if the second calculation result is greater than the memory occupied by the first data block, releasing the memory occupied by the first data block and shifting the storage location of the second data block and the first calculation result forward, so that the starting address of the second data block is the first offset position, and storing the second calculation result continuously after the first calculation result; or, if the second calculation result is greater than the memory occupied by the first data block, storing a portion of the second calculation result in the memory occupied by the first data block, and storing the remaining portion of the second calculation result continuously after the first calculation result.

[0095] For example, combining Figure 5 ,like Figure 7bAs shown, conv2-1 (the second calculation result) is larger than the memory occupied by the first data block (useless data). Therefore, the memory occupied by the first data block can be released, and the storage locations of the second data block and conv1-1 (the first calculation result) can be shifted forward, making the starting address of the second data block the first offset position. conv2-1 (the second calculation result) is then stored consecutively after conv1-1 (the first calculation result). That is, the second calculation result is stored consecutively in memory, and only an offset position and the size of the second calculation result need to be recorded to determine its memory location.

[0096] For example, combining Figure 5 ,like Figure 7c As shown, conv2-1 (the second calculation result) is larger than the memory occupied by the first data block (useless data), so a portion of the data in conv2-1 (such as...) can be used... Figure 7c As shown, conv2-1a) is stored in the memory occupied by the first data block, and another part of the data of conv2-1 (such as conv1-1, the first calculation result) is stored consecutively after conv1-1 (the first calculation result). Figure 7c As shown in conv2-1b). That is, the second calculation result is stored in segments in memory. To determine the memory location of the second calculation result, it is necessary to record the two offset positions, the sizes of conv2-1a and conv2-1b. Relative to... Figure 7b The second calculation result shown is stored contiguously in memory; only an offset position and size need to be recorded. Figure 7c The storage method shown requires recording more information.

[0097] For example, if the first calculation result is useless data when the third calculation result is obtained, the third calculation result reuses the memory occupied by the first calculation result. It is understood that the specific implementation method of the third calculation result reusing the memory occupied by the first calculation result is the same as the specific implementation method of the second calculation result reusing the memory occupied by the first data block, as described above. Please refer to the aforementioned related content; it will not be repeated here.

[0098] For example, combining Figure 5 ,like Figure 7d As shown, taking the example where conv2-1 (the second calculation result) is less than the memory occupied by the first data block, the memory occupied by useless data in the historical calculation results during the calculation of conv1-2 includes the memory occupied by the first data block excluding the memory occupied by conv2-1, as well as the memory occupied by conv1-1. The third calculation result can reuse the memory occupied by useless data in the historical calculation results. Optionally, if conv1-2 is less than or equal to conv1-1, conv1-2 can reuse the memory occupied by conv1-1. Figure 7dThis example illustrates how the memory used by the first calculation result is reused in the third calculation result (conv1-2). Figure 7d The storage method shown allows conv1-2 to be stored consecutively in memory.

[0099] Understandably, in practical applications, the third calculation result can reuse the memory occupied by useless data in the historical calculation results. For the aforementioned third calculation result (conv1-2), it can reuse the memory occupied by the first data block, excluding the memory occupied by conv2-1, and / or the memory occupied by conv1-1. The specific reuse method depends on the size of the third calculation result.

[0100] For example, the fourth calculation result reuses the free memory between storing the second and third calculation results. It is understood that the data stored in this free memory between storing the second and third calculation results is useless data used in calculating the fourth calculation result.

[0101] For example, combining Figure 5 ,like Figure 7e As shown, taking the example where conv2-1 (the second calculation result) is less than the memory occupied by the first data block, the memory occupied by useless data in the historical calculation results during the calculation of conv2-2 (the fourth calculation result) includes the memory occupied by the first data block excluding the memory occupied by conv2-1, as well as the memory occupied by the second data block. The third calculation result can reuse the memory occupied by useless data in the historical calculation results. That is, conv2-2 can reuse the memory between storing conv2-1 and storing conv1-2. Figure 7e The illustration is based solely on the fourth calculation result (conv2-2) being stored consecutively after conv2-1 (the second calculation result).

[0102] Understandably, when calculating and storing M data blocks sequentially from the first to the Kth network layer, the weights of each data block in each network layer can share the weights of the corresponding network layer in the neural network model, thus eliminating the need to retrain the model. Furthermore, the memory between different network layers can be reused when storing the calculation results of a data block in different network layers; similarly, the memory between different data blocks can also be reused when storing the calculation results of different data blocks in the first to Kth network layers. In other words, the neural network model optimization method in this embodiment can achieve memory reuse between layers and between blocks. Compared to the data storage methods in the prior art, this embodiment not only eliminates the need to retrain the model but also significantly reduces memory consumption, enabling the neural network to be deployed in embedded systems with limited memory.

[0103] For example, in step S302 above, when calculating M data blocks layer by layer from the first network layer to the Kth network layer and storing the calculation results in memory, the storage location of each network layer of each data block in memory can be obtained through the following steps a to e:

[0104] Step a: Determine the size of the useless data freebuf when calculating the calculation result of the i-th data block in the j-th network layer, i is greater than or equal to 1 and less than or equal to M, j is greater than or equal to 1 and less than or equal to K, and i and j are integers.

[0105] For example, when calculating the result of the i-th data block in the j-th network layer, the j-th network layer can be referred to as the current network layer. The size of the useless data (freebuf) refers to the size of the data stored before the current network layer and the calculation results that will not be used when calculating the calculation results of the current network layer and the network layers after the current network layer.

[0106] Optionally, the freebuf size for calculating the result of the i-th data block in the j-th network layer can be 0. That is, the data and calculation results stored before the current network layer will be used when calculating the calculation results of the current network layer and the network layers after it.

[0107] For example, when calculating the result of the i-th data block in the j-th network layer, the weight of the i-th data block in the j-th network layer is the same as the weight of the neural network model in the j-th network layer. That is, the weight of the i-th data block in the j-th network layer can share the weight of the neural network model in the j-th network layer. Therefore, this embodiment does not require retraining the model, which saves more time.

[0108] Step b: Determine the size of the computation result of the i-th data block in the j-th network layer (curbuf).

[0109] Step c: If curbuf is less than or equal to freebuf, the calculation result of the i-th data block in the j-th network layer reuses the memory occupied by useless data, and the storage location of the calculation result of the i-th data block in the j-th network layer in memory is recorded.

[0110] Understandably, the specific implementation of reusing the memory occupied by useless data in the calculation result of the i-th data block in the j-th network layer in step c can be found in the relevant description of the foregoing embodiments.

[0111] Step d: If curbuf is greater than freebuf, the calculation result of the i-th data block in the j-th network layer reuses the memory occupied by useless data, and the storage location of the calculation result of the i-th data block in the j-th network layer in the target memory is recorded.

[0112] Understandably, the specific implementation of reusing the memory occupied by useless data in the calculation result of the i-th data block in the j-th network layer in step d can be found in the relevant description of the foregoing embodiments.

[0113] It should be noted that when freebuf is 0, there is no memory available to reuse the calculation result of the i-th data block in the j-th network layer. Therefore, the calculation result of the i-th data block in the j-th network layer can be stored in free memory.

[0114] Step e: Following steps a to d, sequentially determine the memory storage location of the calculation results of the M data blocks in each of the first to the Kth network layers, from the first to the Kth network layers.

[0115] Understandably, following steps a to e above, M data blocks can be computed layer by layer from the 1st to the Kth network layer, and the computation results can be stored sequentially in memory. The storage location of each data block within the computation result of each network layer can then be obtained. Because memory can be reused between layers and between blocks during the execution of steps a to e, memory consumption can be greatly reduced, allowing neural networks to be deployed in embedded systems with limited memory.

[0116] S303. Combine the calculation results of M data blocks in the Kth network layer and use them as the input of the (K+1)th network layer.

[0117] For example, after calculating the calculation results of M data blocks in the first to the Kth network layers, the calculation results of the M data blocks in the Kth network layer can be merged and used as the input of the (K+1)th network layer.

[0118] Optionally, after step S303 above, the calculation may also include: performing calculations layer by layer from the (K+1)th network layer to the Nth network layer.

[0119] For example, such as Figure 5 As shown, the calculation results of the first and second data blocks in the second network layer can be combined and used as the input of the third network layer. The calculation is performed layer by layer from the third to the fifth network layer.

[0120] For example, when merging the calculation results of M data blocks at the Kth network layer, the calculation results of each data block at the Kth layer can be merged along the w dimension to ensure that the data arrangement is consistent with the arrangement of the unsegmented data. For example, as Figure 8 As shown, taking the input data divided into 6 data blocks as an example, the calculation results of the 6 data blocks in the Kth network layer are w1, w2, w3, w4, w5 and w6 respectively. The calculation results of the 6 data blocks in the Kth network layer are merged in the w dimension. The data arrangement after merging is the same as the data arrangement before splitting.

[0121] For example, when the calculation results of M data blocks in the Kth network layer are combined and used as the input of the (K+1)th network layer, and the calculation is performed layer by layer from the (K+1)th network layer to the Nth network layer, the calculation results from the (K+1)th network layer to the Nth network layer can be stored in memory sequentially.

[0122] For example, the computation results of each network layer from the (K+1)th to the Nth network layer can reuse the memory occupied by useless data in the historical computation results. It is understood that the specific method for reusing the memory occupied by useless data in the computation results of each network layer from the (K+1)th to the Nth network layer can be found in step S302, and will not be elaborated further here.

[0123] For example, such as Figure 7e As shown, the data currently stored in memory includes con2-1, conv2-2, a portion of the second data block, and the calculation results of conv1-2. Combined with... Figure 5 As shown, when calculating the results of the third network layer (pooling layer 1), the fourth network layer (pooling layer 2), and the fifth network layer (fully connected layer), the calculation results of con2-1 and conv2-2 are used. However, some data from the second data block and conv1-2 are not used. Therefore, the calculation result of the third network layer (pooling layer 1) can reuse some data from the second data block and the memory occupied by conv1-2.

[0124] Understandably, when calculating and storing the computation results of each network layer from the (K+1)th to the Nth network layer, the current network layer can reuse the memory occupied by useless data in the historical computation results, realizing memory reuse between layers, which can further reduce memory consumption and enable the neural network to be deployed in embedded systems with small memory.

[0125] For example, in step S303 above, when the calculation results of M data blocks in the Kth network layer are merged and used as the input of the (K+1)th network layer, and the calculation is performed layer by layer from the (K+1)th network layer to the Nth network layer and the calculation results are stored in memory sequentially, the storage location of each network layer in memory can be obtained through the following steps f to k:

[0126] Step f: Combine the calculation results of the M data blocks in the Kth network layer and use them as the input of the (K+1)th network layer.

[0127] Step g: Determine the size of the useless data freebuf' when calculating the result of the kth network layer, where k is greater than or equal to K+1 and less than or equal to N, and k is an integer.

[0128] For example, when calculating the result of the k-th network layer, the k-th network layer can be referred to as the current network layer. The size of the useless data 'freebuf' refers to the size of the data stored before the current network layer and the calculation results that will not be used when calculating the calculation results of the current network layer and the network layers after the current network layer.

[0129] Optionally, the size of the useless data (freebuf') used to calculate the result of the k-th network layer can be 0. That is, the data and calculation results stored before the current network layer will be used when calculating the calculation results of the current network layer and the network layers after it.

[0130] Step h: Determine the size of the calculation result of the k-th network layer, curbuf'.

[0131] Step i: If curbuf' is less than or equal to freebuf', reuse the memory occupied by useless data for the calculation result of the k-th network layer, and record the storage location of the calculation result of the k-th network layer in memory.

[0132] Understandably, the specific implementation of reusing the memory occupied by useless data in the calculation result of the kth network layer in step i can be found in the relevant description of the foregoing embodiments.

[0133] Step j: If curbuf' is greater than freebuf', reuse the memory occupied by useless data for the calculation result of the k-th network layer, and record the storage location of the calculation result of the k-th network layer in the target memory.

[0134] Understandably, the specific implementation of reusing the memory occupied by useless data in the calculation result of the kth network layer in step j can be found in the relevant description of the foregoing embodiments.

[0135] It should be noted that when freebuf' is 0, there is no memory available to reuse the calculation result of the k-th network layer. Therefore, the calculation result of the k-th network layer can be stored in free memory.

[0136] Step k: Following steps g to j, determine the memory storage location of the calculation results of each network layer from the (K+1)th to the Nth network layer.

[0137] Understandably, following steps f to k above, the computation is performed layer by layer from the (K+1)th to the Nth network layer, and the results are stored sequentially in memory. The storage location of the computation result for each network layer is then obtained. Because memory can be reused between layers during the execution of steps f to k, memory consumption is significantly reduced, allowing neural networks to be deployed in embedded systems with limited memory.

[0138] The neural network model optimization method provided in this application divides the input data of the neural network model into M data blocks along the width w dimension. The M data blocks are then sequentially calculated layer by layer, from the 1st to the Kth network layer. The weights of each data block in each network layer can share the weights of the corresponding network layer in the neural network model. The calculation results of the M data blocks in the Kth network layer are then combined and used as the input to the (K+1)th network layer. In this embodiment, the neural network model optimization method allows each data block to share the weights of the corresponding network layer in the neural network model, thus eliminating the need for model retraining and saving time. Furthermore, it enables memory reuse between layers and between blocks, thereby reducing the memory consumption of embedded systems with limited memory when running the model.

[0139] This application also provides an optimization method for a neural network model, such as... Figure 9 As shown, steps S304-S306 may be included before step S301.

[0140] S304. Determine the merged layer as the Kth network layer.

[0141] Among them, under the premise of reusing memory, the memory occupied by storing the calculation results of the Kth network layer and the data before the calculation results of the Kth network layer is the peak memory max_tmpbuf. The peak memory is the maximum memory that can be occupied by all existing calculation results of the neural network model; the calculation results of the (K+1)th network layer to the Nth network layer reuse the peak memory.

[0142] For example, Figure 10As shown, the specific method for determining the merging layer in step S305 above can be achieved by obtaining the position of the merging layer through the following steps a1 to g1:

[0143] Step a1: Determine the memory occupied by the input data.

[0144] Step b1: Initialize peak memory max_tmpbuf as the memory size occupied by the input data.

[0145] Step c1: Determine the size of the useless data when calculating the result of the m-th network layer, where m is greater than or equal to 1 and less than or equal to N, and m is an integer.

[0146] For example, when calculating the result of the m-th network layer, the m-th network layer can be referred to as the current network layer. The size of the useless data freebuf1 refers to the size of the data stored before the current network layer and the calculation results that will not be used when calculating the calculation results of the current network layer and the network layers after the current network layer.

[0147] Optionally, the size of the useless data (freebuf1) used to calculate the result of the m-th network layer can be 0. That is, the data and calculation results stored before the current network layer will be used when calculating the calculation results of the current network layer and the network layers after it.

[0148] Step d1: Determine the size of the calculation result of the m-th network layer (curbuf1).

[0149] Step e1: If curbuf1 is less than or equal to freebuf1, the calculation result of the m-th network layer reuses the memory occupied by freebuf1, and max_tmpbuf remains unchanged.

[0150] Understandably, the specific implementation of reusing the memory occupied by useless data in the calculation result of the m-th network layer in step e1 can be found in the relevant description of the foregoing embodiments.

[0151] Step f1: If curbuf1 is greater than freebuf1, the calculation result of the m-th network layer reuses the memory occupied by freebuf1, and the first value is added to max_tmpbuf, which is the difference between curbuf1 and freebuf1. The value of the merged layer K is determined to be m.

[0152] Understandably, the specific implementation of reusing the memory occupied by the useless data (freebuf1) of the calculation result of the m-th network layer in step f1 can be found in the relevant description of the foregoing embodiments.

[0153] It should be noted that when freebuf1 is 0, there is no memory available to reuse the calculation result of the m-th network layer. Therefore, the calculation result of the m-th network layer can be stored in free memory.

[0154] Optionally, in step f1 above, if curbuf1 is greater than freebuf1, the calculation result of the m-th network layer can also be stored in free memory instead of reusing the memory occupied by freebuf1. That is, the first value to be added to max_tmpbuf is curbuf1. It can be understood that reusing the memory occupied by freebuf1 in step f1 above and adding the difference between curbuf1 and freebuf1 to max_tmpbuf is more memory-efficient than this implementation method.

[0155] Step g1: Following steps c1 to f1, traverse from the first network layer to the Nth network layer to determine the size of the peak memory max_tmpbuf and the value of the merging layer K.

[0156] Understandably, following steps a1 to g1 above, calculations can be performed layer by layer from the first to the Nth network layer, and the results can be stored sequentially in memory. Furthermore, when storing the current network layer's calculation result in memory, the memory occupied by useless data from previous calculations can be reused. Therefore, after traversing from the first to the Nth network layer, the size of the peak memory `max_tmpbuf` and the corresponding layer depth (i.e., the merging layer K) can be obtained. The size of this peak memory `max_tmpbuf` is the maximum memory that can be occupied by all existing calculation results of the neural network model, and the calculation results from the (K+1)th to the Nth network layer all reuse this peak memory.

[0157] S305. Calculate the sum of memory occupied by the calculation results from the (K+1)th network layer to the Nth network layer using addbuf.

[0158] For example, step S305 above can calculate the sum of memory occupied by the calculation results of the (K+1)th network layer to the Nth network layer, addbuf, while reusing memory.

[0159] For example, based on steps a1 to f1 in step S304 above, the network can be traversed sequentially from the (K+1)th network layer to the Nth network layer to determine the sum of memory occupied by the calculation results from the (K+1)th network layer to the Nth network layer, addbuf.

[0160] S306. Determine the number of blocks M based on peak memory max_tmpbuf and addbuf.

[0161] For example, the above method of determining the number of partitioned blocks M based on peak memory max_tmpbuf and addbuf can be... The number of blocks is determined to be M. This indicates rounding down from max_tmpbuf / addbuf.

[0162] Understandably, the neural network model optimization method in this embodiment can determine the merging layer K and the number of segments M based on the peak memory max_tmpbuf and addbuf, while reusing memory. The input data is then divided into M data blocks along the w dimension. Each of the M data blocks is calculated layer by layer from the 1st to the Kth layer, where the weights of each data block in each layer can share the weights of the corresponding layer in the neural network model. The calculation results of the M data blocks in the Kth layer are merged and used as the input to the (K+1)th layer. In this embodiment, the neural network model optimization method allows each data block to share the weights of the corresponding layer in the neural network model, thus eliminating the need for model retraining and saving time. Furthermore, it enables memory reuse between layers and between blocks, significantly reducing memory consumption and allowing the neural network to be deployed in embedded systems with limited memory.

[0163] This application also provides an optimization method for a neural network model, such as... Figure 11 As shown, steps S307-S308 may also be included after step S306 and before step S301.

[0164] S307. Based on the number of segments M and the merging layer K, determine the first mean average precision (mAP).

[0165] In this model, the weights of each data block in the nth network layer are the same as the weights of the nth network layer in the neural network model. n is greater than or equal to 1 and less than or equal to K, and n is an integer.

[0166] The mean accuracy (mAP) mentioned above is a performance metric used to evaluate the model.

[0167] For example, when determining the first average precision based on the number of segments M and the merging layers K, a new prototxt can be generated according to the number of segments M and the merging layers K, wherein the weights of the M data blocks in the first to the kth network layers can share the weights of the neural network model in the corresponding network layers; and the first mAP is determined.

[0168] Optionally, if the difference between the first average precision and the second mAP when the input data is not segmented is less than or equal to a preset threshold, step S308 does not need to be executed, and the aforementioned steps S301-S303 can be executed directly using the number of segmented blocks M and the merging layer K. If the difference between the first average precision and the second mAP when the input data is not segmented is greater than the preset threshold, step S308 continues to be executed.

[0169] S308. If the difference between the first average precision mean and the second mAP when the input data is not segmented is greater than a preset threshold, the preset algorithm is used to update the merging layer K to K' and the number of segments M to M'.

[0170] Where M' is less than M, and K' is less than K.

[0171] For example, the above-mentioned preset algorithm can be a bisection method. That is, when the mAP after segmentation differs significantly from the mAP before segmentation, a bisection method can be used to determine a new merging layer. It can be understood that the new merging layer K' is less than K, that is, the segmentation is performed on a network layer K' that is shallower than the Kth network layer.

[0172] Optionally, after determining the new merging layer K' using the binary search method, and under the premise of memory reuse, the peak memory max_tmpbuf' is determined by sequentially traversing from the first network layer to the K' network layer according to steps a1 to f1 in step S304; and the sum of memory occupied by the calculation results from the K'+1 network layer to the Nth network layer is determined by addingbuf' according to steps a1 to f1 in step S304. The new number of partitioned blocks M' is then determined based on max_tmpbuf' and addbuf'. For example, the number of partitioned blocks M' can be determined by... The number of blocks to be divided is determined to be M'.

[0173] Optionally, after step S308, step S307 can be executed further. This involves determining a new first mAP based on the number of segments M' and the merging layer K', and further determining whether the difference between this new first mAP and the second mAP before data segmentation is less than a preset threshold. If less, M' is determined as the number of segments M, and K' is determined as the merging layer K, and steps S301-S303 are executed. If greater, a binary search method is used to update the merging layer K to K" (segmentation is performed at a shallower layer), and the number of segments M is updated to M" (where M" is less than M', and K" is less than K'). Steps S307 and S308 are then executed until the difference between the first mAP and the second mAP is less than the preset threshold. In other words, the segmented mAP and the unsegmented mAP are essentially the same.

[0174] For example, Figure 12As shown, generating a new prototxt based on the merging layer K and the number of partitioned blocks M, and sharing parameters among the M data blocks, reveals that the overlap effect of shallow merging and partitioning, as well as the loss due to the precision of shared parameters among partitioned blocks, is minimal. Figure 12 As shown, the mAP values ​​of the unsegmented and segmented layers are basically the same. Therefore, when the mAP after segmentation differs significantly from the unsegmented mAP, a bisection method can be used to segment the layers at a shallow depth to ensure that the accuracy remains consistent between the segmented and unsegmented layers.

[0175] Understandably, in this embodiment of the application, when the input data is divided into M data blocks, since the M data blocks are calculated sequentially from the first network layer to the Kth network layer, the more blocks there are, the more it affects the hardware's computing performance. Therefore, in this application, under the premise of memory reuse, the memory occupied by the calculation results of all data blocks in the network layers before the merging layer is greater than or equal to the memory occupied by the calculation results in the network layers after the merging layer. This ensures that the hardware's computing performance is not significantly affected and also reduces memory consumption.

[0176] For example, with Figure 5 Taking the segmentation scheme shown as an example, after passing through the first and second network layers, which are the two network layers with the highest memory consumption, the network is merged. The memory usage is compared between using the segmentation scheme and not using the segmentation scheme. Figure 13a The image shows the input data, memory usage of the first and second network layers before splitting. Figure 13b The diagram shows the memory usage of the input data, the first network layer, and the second network layer after memory is segmented and reused, according to an embodiment of this application. Figure 13a and Figure 13b As shown, the partitioning scheme of this application, through memory reuse between layers and blocks, occupies far less memory than the unpartitioned scheme, thus significantly reducing memory consumption. This partitioning scheme solves the problem of significant memory waste caused by the uneven memory distribution of the inverted cone structure in the prior art, thus enabling deep learning to be deployed in embedded systems with limited memory.

[0177] This embodiment compares the segmented mAP with the unsegmented mAP. When the difference between the segmented and unsegmented mAP is significant, the merging layer is updated to a shallower network layer. This ensures that the segmented mAP is essentially the same as the unsegmented mAP, thus guaranteeing no loss of accuracy between the segmented and unsegmented states. Then, by executing steps S301-S303 based on the updated number of segments and the merging layer, memory consumption can be further reduced while maintaining accuracy, allowing the neural network to be deployed in embedded systems with limited memory.

[0178] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to implement the above functions, a computer includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in a combination of hardware and computer software. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] This application embodiment can divide the execution entity into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0180] Figure 14 A possible structural schematic diagram of an optimization device for a neural network model is shown. The optimization device 1400 may include: a segmentation unit 1401, a processing unit 1402, and a merging unit 1403.

[0181] For example, the above-described segmentation unit 1401 can be used to perform, for example... Figure 3 Step S301 in the above, and / or other processes used in the technology described herein. The processing unit 1402 described above is used to perform, for example... Figure 3 Step S302, or Figure 9 S304-S306 in, or Figure 10 The steps shown, or Figure 11 The S307-S308 shown, and / or other processes used in the technology described herein. The aforementioned merging unit 1403 is used to perform, for example... Figure 3 Step S303 in the above method embodiments, and / or other processes used in the technology described herein. All relevant content regarding each step in the above method embodiments can be referenced to the functional description of the corresponding functional module, and will not be repeated here.

[0182] Figure 15 This is a schematic diagram illustrating a possible structure of an optimization device 1500 for another neural network model. The optimization device 1500 may include a processor 1501. Optionally, it may also include a memory 1502.

[0183] The memory 1502 is used to store the program code and data of the device, and the processor 1501 is used to process... Figure 15 The operation of the optimization device 1500 for the illustrated neural network model is controlled and managed. For example, the processor 1501 is used to perform, for example... Figure 3 Step S302, or Figure 9 S304-S306 in, or Figure 10 The steps shown, or Figure 11 S307-S308 shown, and / or other processes used in the techniques described herein.

[0184] Optional, Figure 15 The device shown may also include a communication interface 1503, which is used to support the optimization device 1500 of the neural network model to communicate with other devices.

[0185] The processor 1501 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a processing chip, a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various logic blocks, modules, and circuits described in connection with the embodiments of this application. The processor 1501 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The communication interface 1503 can be a transceiver, transceiver circuitry, or transceiver interface, etc. The memory 1502 can be volatile memory or non-volatile memory, etc.

[0186] Optional, Figure 15 The illustrated device may also include a bus 1504. For example, the communication interface 1503, processor 1501, and memory 1502 are interconnected via bus 1504; bus 1504 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1504 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 15 The bus is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. Optionally, the memory 1502 may be included in the processor 1501.

[0187] This application also provides an apparatus, which exists in the form of a chip. The apparatus includes a processor, and optionally, a memory. The memory stores computer-executed instructions. The processor executes the computer-executed instructions to support the implementation of the apparatus. Figure 3 , Figure 9 , Figure 10 or Figure 11 An optimization method for the neural network model in any of the embodiments. Exemplarily, the device can be deployed in an embedded system. The processor described above can be a coprocessor or other processor deployed in the embedded system.

[0188] This application also provides an apparatus that can exist in the form of a chip. The apparatus includes a processor and an interface circuit. The processor communicates through the interface circuit, enabling the apparatus to perform the aforementioned operations. Figure 3 , Figure 9 , Figure 10 or Figure 11 An optimization method for the neural network model in any of the embodiments.

[0189] This application also provides a computer-readable storage medium storing computer program code. When a computer or processor executes the computer program code, an electronic device performs... Figure 3 , Figure 9 , Figure 10 or Figure 11 An optimization method for the neural network model in any of the embodiments.

[0190] This application also provides a computer program product that, when run on a computer or processor, causes the computer or processor to perform... Figure 3 , Figure 9 , Figure 10 or Figure 11 An optimization method for the neural network model in any of the embodiments.

[0191] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to implement the above functions, a computer includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in a combination of hardware and computer software. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0192] The steps of the methods or algorithms described in this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a core network interface device. Of course, the processor and storage medium can also exist as discrete components in the core network interface device.

[0193] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0194] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A method for optimizing a neural network model, characterized in that, Applied to the field of image recognition, the neural network model includes N network layers, where N is greater than 2 and N is an integer. The method includes: The input data of the neural network model is divided into M data blocks, where M is greater than or equal to 2, M is an integer, and the input data is an image; The M data blocks are sequentially calculated layer by layer from the first network layer to the Kth network layer, where K is greater than 1 and less than N, and K is an integer; wherein, the weight corresponding to the i-th network layer of each data block is the same as the weight corresponding to the i-th network layer in the neural network model, where i is greater than or equal to 1 and less than or equal to K, and i is an integer; The calculation results of the M data blocks in the Kth network layer are combined and used as the input of the (K+1)th network layer; The merged layer is determined to be the Kth network layer; Wherein, under the premise of reusing memory, the memory occupied by storing the calculation result of the Kth network layer and the data before the calculation result of the Kth network layer is the peak memory max_tmpbuf, and the peak memory is the maximum memory that can be occupied by all existing calculation results of the neural network model; the calculation results of the (K+1)th network layer to the Nth network layer reuse the peak memory.

2. The method according to claim 1, characterized in that, The dimensions of the input data include width w, height h, and depth c. Dividing the input data of the neural network model into M data blocks includes: The input data of the neural network model is divided into M data blocks along the width w dimension.

3. The method according to claim 1 or 2, characterized in that, The calculation result of the Pth data block at level J reuses the memory occupied by useless data in the historical calculation results, where P is any positive integer greater than or equal to 1 and less than or equal to M, and J is any positive integer greater than 1 and less than or equal to K.

4. The method according to claim 1 or 2, characterized in that, The M data blocks include a first data block and a second data block, and the K network layers include a first network layer and a second network layer. The calculation result of the first data block in the first network layer is a first calculation result, the calculation result of the first data block in the second network layer is a second calculation result, the calculation result of the second data block in the first network layer is a third calculation result, and the calculation result of the second data block in the second network layer is a fourth calculation result. The starting address of the first data block is a first offset position, the starting address of the second data block is a second offset position, and the starting address of the first calculation result is a third offset position. If the second calculation result is obtained, and the first data block is useless data, the second calculation result reuses the memory occupied by the first data block.

5. The method according to claim 4, characterized in that, The second calculation result reuses the memory occupied by the first data block, specifically including: If the second calculation result is less than or equal to the memory occupied by the first data block, the second calculation result is stored in the memory with the first offset position as the starting address; If the second calculation result is greater than the memory occupied by the first data block, the memory occupied by the first data block is released, and the storage locations of the second data block and the first calculation result are shifted forward, so that the starting address of the second data block is the first offset position, and the second calculation result is stored continuously after the first calculation result; or, If the second calculation result is greater than the memory occupied by the first data block, a portion of the second calculation result is stored in the memory occupied by the first data block, and the remaining portion of the second calculation result is stored continuously after the first calculation result.

6. The method according to claim 4, characterized in that, The method further includes: If the first calculation result is useless data when the third calculation result is obtained, the third calculation result reuses the memory occupied by the first calculation result; The fourth calculation result is reused to store the free memory between the second calculation result and the third calculation result.

7. The method according to claim 1, characterized in that, The method further includes: The sum of memory usage for the computation results from the (K+1)th network layer to the Nth network layer is calculated using the variable `addbuf`. Will The number of blocks to be divided is determined to be M; where, This indicates rounding down from max_tmpbuf / addbuf.

8. The method according to claim 7, characterized in that, The method further includes: Based on the number of data blocks M and the merging layer, the first average precision mAP is determined; wherein, the weight of each data block in the nth network layer is the same as the weight of the nth network layer in the neural network model, where n is greater than or equal to 1 and less than or equal to K, and n is an integer; If the difference between the first average precision mean and the second average precision mean mAP when the input data is not segmented is greater than a preset threshold, a preset algorithm is used to update the number of segments M to M' and the merging layer K to K', where M' is less than M and K' is less than K; the average precision mean mAP is a performance metric used to evaluate the model.

9. An optimization device for a neural network model, characterized in that, Applied to the field of image recognition, the neural network model includes N network layers, where N is greater than 2 and N is an integer. The device includes: A segmentation unit is used to segment the input data of the neural network model into M data blocks, where M is greater than or equal to 2, M is an integer, and the input data is an image; The processing unit is configured to sequentially calculate the M data blocks layer by layer, from the first network layer to the Kth network layer, where K is greater than 1 and less than N, and K is an integer; wherein the weight corresponding to the i-th network layer of each data block is the same as the weight corresponding to the i-th network layer in the neural network model, where i is greater than or equal to 1 and less than or equal to K, and i is an integer; The merging unit is used to merge the calculation results of the M data blocks in the Kth network layer and use them as the input of the (K+1)th network layer. The processing unit is further configured to: determine the merging layer as the Kth network layer; wherein, under the premise of reusing memory, the memory occupied by storing the calculation results of the Kth network layer and the data before the calculation results of the Kth network layer is the peak memory max_tmpbuf, and the peak memory is the maximum memory that can be occupied by all existing calculation results of the neural network model; the calculation results of the (K+1)th network layer to the Nth network layer reuse the peak memory.

10. The apparatus according to claim 9, characterized in that, The dimensions of the input data include width w, height h, and depth c; the segmentation unit is specifically used to segment the input data of the neural network model into M data blocks along the width w dimension.

11. The apparatus according to claim 9 or 10, characterized in that, The calculation result of the Pth data block at level J reuses the memory occupied by useless data in the historical calculation results, where P is any positive integer greater than or equal to 1 and less than or equal to M, and J is any positive integer greater than 1 and less than or equal to K.

12. The apparatus according to claim 9 or 10, characterized in that, The M data blocks include a first data block and a second data block, and the K network layers include a first network layer and a second network layer. The calculation result of the first data block in the first network layer is a first calculation result, the calculation result of the first data block in the second network layer is a second calculation result, the calculation result of the second data block in the first network layer is a third calculation result, and the calculation result of the second data block in the second network layer is a fourth calculation result. The starting address of the first data block is a first offset position, the starting address of the second data block is a second offset position, and the starting address of the first calculation result is a third offset position. The processing unit is specifically configured to, if the processing unit determines that the first data block is useless data when the second calculation result is obtained, reuse the memory occupied by the first data block in the second calculation result.

13. The apparatus according to claim 12, characterized in that, The processing unit is specifically used for: If the second calculation result is less than or equal to the memory occupied by the first data block, the second calculation result is stored in the memory with the first offset position as the starting address; If the second calculation result is greater than the memory occupied by the first data block, the memory occupied by the first data block is released, and the storage locations of the second data block and the first calculation result are shifted forward, so that the starting address of the second data block is the first offset position, and the second calculation result is stored continuously after the first calculation result; or, If the second calculation result is greater than the memory occupied by the first data block, a portion of the second calculation result is stored in the memory occupied by the first data block, and the remaining portion of the second calculation result is stored continuously after the first calculation result.

14. The apparatus according to claim 12, characterized in that, The processing unit is further configured to: If the processing unit determines that the third calculation result is obtained, and the first calculation result is useless data, the third calculation result reuses the memory occupied by the first calculation result; The fourth calculation result is reused to store the free memory between the second calculation result and the third calculation result.

15. The apparatus according to claim 9, characterized in that, The processing unit is further configured to: The sum of memory usage for the computation results from the (K+1)th network layer to the Nth network layer is calculated using the variable `addbuf`. Will The number of blocks to be divided is determined to be M; where, This indicates rounding down from max_tmpbuf / addbuf.

16. The apparatus according to claim 15, characterized in that, The processing unit is further configured to: Based on the number of data blocks M and the merging layer, the first average precision mAP is determined; wherein, the weight of each data block in the nth network layer is the same as the weight of the nth network layer in the neural network model, where n is greater than or equal to 1 and less than or equal to K, and n is an integer; If the difference between the first average precision mean and the second average precision mean mAP when the input data is not segmented is greater than a preset threshold, a preset algorithm is used to update the number of segments M to M' and the merging layer K to K', where M' is less than M and K' is less than K; the average precision mean mAP is a performance metric used to evaluate the model.

17. An optimization device for a neural network model, characterized in that, The device includes a processor on which the neural network model is deployed. The neural network model includes N network layers, where N is greater than 2 and N is an integer. The processor is used to call program instructions in memory to execute the optimization method of the neural network model as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform an optimization method for a neural network model as described in any one of claims 1-8.

Images