Convolutional neural network optimization method, device and equipment for storage and calculation integrated chip
By optimizing the convolutional neural network method of in-memory computing chips, the problem of excessive cache space is solved by caching only the input data of the smallest computing unit and reusing the data, thus achieving cost and area savings.
Patent Information
- Application Number
- CN202511210143.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-27
- Publication Date
- 2025-11-18
AI Technical Summary
The excessive cache space required for convolutional neural networks in existing in-memory computing chips leads to increased chip cost and area.
By determining the minimum computational unit and the target number of operations for each computational network layer, only the input data required by the minimum computational unit is cached, and the cached data is updated and deleted during the operation. The computational network layer is segmented to reduce the cache requirement, and data reuse processing is performed between adjacent layers.
It effectively reduces cache usage, saves cost and area of in-memory computing chips, and improves data transfer efficiency.
Smart Images
Figure CN120975153A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of neural networks, and in particular to a convolutional neural network optimization method, device and equipment for a memory-compute integrated chip. BACKGROUND
[0002] The memory-compute integrated chip can store the weights of a convolutional neural network model and the like in a memory-compute array, but there are still a large amount of computing intermediate data that need to be stored in a cache in the function calculation of each network layer of the convolutional neural network. In the prior art, the complete output of a previous layer in the convolutional neural network is the input of a next layer, and the complete input and output of each layer need to be stored in the cache. The space of the cache is determined by the data capacity of different levels of the convolutional neural network. Therefore, the space of the cache needs to be at least more than the data capacity of the largest layer (the largest layer refers to a layer level with the largest sum of input data and output data in the complete cache), and the larger the data capacity of the largest layer in the convolutional neural network, the higher the requirement for the storage space capacity of the cache. Since these cache data are stored in a running memory SRAM, the chip cost and the chip area size are greatly increased.
[0003] Therefore, there is an urgent need for a convolutional neural network optimization method for a memory-compute integrated chip to reduce the problem of large cache occupation. SUMMARY
[0004] In view of the above problems, the present application is proposed in order to provide a convolutional neural network optimization method for a memory-compute integrated chip to overcome the above problems or at least partially solve the above problems.
[0005] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.
[0006] According to a first aspect of an embodiment of the present application, a convolutional neural network optimization method for a memory-compute integrated chip is provided, applied to a memory-compute integrated chip, and the convolutional neural network optimization method for the memory-compute integrated chip comprises:
[0007] determining a minimum calculation unit corresponding to each operation network layer and a target operation number respectively;
[0008] reading, from the cache array, previous input data required by the minimum calculation unit of a previous network layer, inputting the read previous input data into the previous network layer, writing, after the previous network layer performs operation on the previous input data to obtain previous output data, the previous output data into the cache array and updating an actual operation number of the previous network layer; the previous network layer refers to an operation network layer currently performing operation in the target memory-compute integrated chip;
[0009] when the actual operation number is less than the target operation number, updating the previous input data in the cache array and continuing to determine the previous output data corresponding to the updated previous input data and continuing to update the actual operation number of the previous network layer, until the actual operation number of the previous network layer is equal to the target operation number of the previous network layer, and determining that the previous network layer completes operation;
[0010] detecting whether the previous network layer is the last operation network layer of the storage-computation integrated chip;
[0011] when the previous network layer is the last operation network layer in the storage-computation integrated chip, determining the output result of the storage-computation integrated chip based on all the previous output data in the cache array.
[0012] According to a second aspect of the embodiment of the present application, a convolutional neural network optimization device for a storage-computation integrated chip is provided, which comprises:
[0013] a data operation module, configured to determine the minimum calculation unit and the target operation number corresponding to each operation network layer respectively, read the previous input data required by the minimum calculation unit of the previous network layer from the cache array, input the read previous input data into the previous network layer, so as to obtain the previous output data after the previous input data is executed in the previous network layer, write the previous output data into the cache array, and update the actual operation number of the previous network layer; the previous network layer refers to the operation network layer currently executing operation in the target storage-computation integrated chip;
[0014] a data update module, configured to, when the actual operation number is less than the target operation number, update the previous input data in the cache array and continue to determine the previous output data corresponding to the updated previous input data and continue to update the actual operation number of the previous network layer, until the actual operation number of the previous network layer is equal to the target operation number of the previous network layer, and determine that the previous network layer completes operation;
[0015] a data output module, configured to detect whether the previous network layer is the last operation network layer of the storage-computation integrated chip; when the previous network layer is the last operation network layer in the storage-computation integrated chip, determine the output result of the storage-computation integrated chip based on all the previous output data in the cache array.
[0016] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a processor and a memory, the memory storing computer program instructions capable of being executed by the processor, and the processor executes the computer program instructions to implement the instructions of the method according to any one of the above.
[0017] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
[0018] The method for optimizing a convolutional neural network for an integrated storage and computing chip provided in the embodiments of the present application determines the preceding input data required by the minimum calculation unit of each operation network layer in the convolutional neural network, and only caches the preceding input data required by the minimum calculation unit of each operation network layer when performing operation, which greatly reduces the cache occupation and saves the cost and area size of the integrated storage and computing chip.
[0019] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0021] Figure 1 A flowchart of a method for optimizing a convolutional neural network for an integrated storage and computing chip is provided in the embodiments of the present application.
[0022] Figure 2 A flowchart of data operation of adjacent two operation network layers is provided.
[0023] Figure 3 A flowchart of splitting the operation network layer in the integrated storage and computing chip is provided.
[0024] Figure 4 A flowchart of data multiplexing of the preceding network layer is provided.
[0025] Figure 5 A principle structure diagram of a device for optimizing a convolutional neural network for an integrated storage and computing chip is provided in the embodiments of the present application. DETAILED DESCRIPTION
[0026] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings.
[0027] Various structural diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These diagrams are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and precision, and certain details can be omitted. The shapes of various regions, layers, and the relative size and positional relationship therebetween shown in the diagrams are merely exemplary, and in actuality, they can be deviated due to manufacturing tolerances or technical limitations, and regions / layers with different shapes, sizes, and relative positions can be additionally designed by those skilled in the art according to actual needs.
[0028] In the context of the present disclosure, when a layer / element is referred to as being located "on" another layer / element, the layer / element can be directly located on the other layer / element, or there can be an intermediate layer / element therebetween. In addition, if a layer / element is located "on" another layer / element in one orientation, it can be located "under" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components can be denoted by the same or similar reference numerals.
[0029] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with specific embodiments. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical solutions of the present disclosure, rather than limitations of the technical solutions of the present disclosure. In the case of no conflict, the technical features in the embodiments of the present disclosure and the embodiments can be combined with each other.
[0030] Figure 1 is a flowchart of a convolutional neural network optimization method for a memory-compute integrated chip provided by an embodiment of the present disclosure, as shown in Figure 1 The convolutional neural network optimization method for the memory-compute integrated chip includes the following steps:
[0031] S1, respectively determining a minimum calculation unit corresponding to each of the operation network layers and a target operation number;
[0032] In an embodiment of the present disclosure, the convolutional neural network is embedded on the memory-compute integrated chip, the convolutional neural network includes a plurality of operation network layers, and the memory-compute integrated chip further includes a cache array. The operation network layers are all used for data operation, for example, convolution operation. The memory-compute integrated chip can map and store weight data (for example, a convolution weight matrix) of the convolutional neural network in a memory-compute array, so that the weight parameters do not need to be accessed during operation, thereby improving the calculation speed. Correspondingly, input data and output data and the like of the operation network layers during operation are cached in the cache array.
[0033] The operation network layer can be, for example, a convolution layer, a pooling layer, a fully connected layer, a normalization and dropout layer, an activation function, and the like. The level of the convolutional neural network can be determined according to actual requirements, and embodiments of the present application do not limit this. The cache array can be, for example, a running memory SRAM or other storage components with the same function.
[0034] The minimum calculation unit is determined according to the corresponding convolution kernel of each operation network layer. For example, when the size of the convolution kernel is 3x3, the dimensions of the input data and the output data of the corresponding minimum calculation unit are 3x3. The target operation number is the number of calculations required for the minimum calculation unit of each operation network layer to complete the entire calculation process on the previous input data. The target operation number is determined according to the dimension size of the input data of each operation network layer, the size of the convolution kernel, and the step length of the convolution kernel movement. For example, when the dimension size of the previous input data is 3x6, the size of the convolution kernel is 3x3, and the movement step length of the convolution kernel is 1, the target operation number is 5.
[0035] S2, reading the previous input data required by the minimum calculation unit of the previous network layer from the cache array, inputting the read previous input data into the previous network layer, and writing the previous output data into the cache array and updating the actual operation number of the previous network layer after performing operation on the previous input data in the previous network layer to obtain the previous output data. The previous network layer refers to the operation network layer currently performing operation in the target memory-computing integrated chip
[0036] When the input data of any one of the operation network layers is input during the operation of the operation network layer, not all input data is directly processed, but multiple operations are performed based on the amount of data required for operation. When all input data and output data during each operation are completely cached, the cache space will be seriously occupied. In order to reduce the cache occupation, the previous input data required by the minimum calculation unit corresponding to each operation network layer is cached in the embodiments of the present application. The previous output data is obtained after each operation through the minimum calculation unit, and the actual operation number of the previous network layer is updated. When the actual operation number is less than the target operation number, the data cached in the cache array is the previous input data and the previous output data of the operation network layer.
[0037] S3, when the actual operation number is less than the target operation number, updating the previous input data in the cache array and continuing to determine the previous output data corresponding to the updated previous input data and continuing to update the actual operation number of the previous network layer until the actual operation number of the previous network layer is equal to the target operation number of the previous network layer, and determining that the previous network layer completes operation;
[0038] When the actual operation number is less than the target operation number, the embodiment of the application updates the previous input data, the previous output data and the actual operation number in the cache array every time the minimum calculation unit performs a function operation, wherein the previous input data is updated by synchronously deleting the previously cached previous input data after the new previous input data is inputted, and the previous output data is cached.
[0039] When the actual operation number of the previous network layer is equal to the target operation number of the previous network layer, it can be considered that the previous network layer completes the entire operation of the input data, at this time, the embodiment of the application synchronously updates the previous input data, that is, deletes all the previous input data in the cache, at this time, the cache array only caches the previous output data as the previous input data of the subsequent network layer.
[0040] S4, detecting whether the previous network layer is the last operation network layer of the memory-compute integrated chip;
[0041] Referring to Figure 2 FIG. 2 shows a flowchart of data operation of adjacent two operation network layers, since the last operation network layer of the memory-compute integrated chip has no subsequent network layer, the embodiment of the application detects whether the previous network layer is the last operation network layer of the memory-compute integrated chip, after detecting whether the previous network layer is the last operation network layer of the memory-compute integrated chip, the method further comprises the following steps:
[0042] S41, when the previous network layer is not the last operation network layer of the memory-compute integrated chip, if it is detected that there is updated previous output data in the cache array, detecting whether the previous output data in the cache array meets the minimum data amount required by the minimum calculation unit of the subsequent network layer; the subsequent network layer refers to the operation network layer adjacent to the previous network layer and after the previous network layer;
[0043] S42, when the preceding output data in the cache array meets the minimum data required by the minimum calculation unit of the subsequent network layer, the preceding output data required by the minimum calculation unit of the subsequent network layer is selected from the cache array as the subsequent input data, the acquired subsequent input data is input into the subsequent network layer, after the subsequent network layer performs operation on the subsequent input data to obtain the subsequent output data, the subsequent output data is written into the cache array, and the actual operation number of the subsequent network layer is updated, until the actual operation number of the subsequent network layer is equal to the target operation number of the subsequent network layer, it is determined that the operation of the subsequent network layer is completed;
[0044] S43, whether the subsequent network layer is the last operation network layer of the storage-computing integrated chip is determined, when the subsequent network layer is not the last operation network layer of the storage-computing integrated chip, the operation network layer adjacent to the subsequent network layer and after the subsequent network layer is determined as a new subsequent network layer, and the subsequent network layer is taken as a new preceding network layer, and the operation process of the new subsequent network layer is continued to be completed.
[0045] S44, until the new subsequent network layer is the last operation network layer of the storage-computing integrated chip, the output result of the storage-computing integrated chip is determined according to the subsequent output data of the new subsequent network layer which is the last operation network layer.
[0046] The embodiment of the application takes the first layer operation network layer of the convolutional neural network as a convolutional layer and the second layer operation network layer as a pooling layer as an example, specifically, in the convolutional layer, if the size of the convolution kernel is 3x3, the preceding input data cached in the convolutional layer at each convolution is a value with an input data amount of 3x3 size, and one output result (i.e. preceding output data) is obtained by convolution, when multiple convolutions are performed, multiple output results are cached, if the preceding input data required by the minimum calculation unit of the pooling layer when performing operation each time needs 4 values, when 4 output results are cached in the convolutional layer, the 4 output results are taken as the input data amount of the pooling layer to perform functional operation, and the data in the cache is updated, i.e. the 4 output results output by the convolutional layer are deleted, to obtain the pooling operation result of the functional operation of the pooling layer; when the number of the pooling operation results corresponding to the cache in the pooling layer meets the basic operation requirement of the next layer operation network layer (i.e. the preceding input data required by the minimum calculation unit of the operation network layer), the pooling operation result of the preceding output data is taken as the preceding input data of the next layer operation network layer to perform operation, and the same is true for the last layer operation network layer, until the preceding output data of the last layer operation network layer meets the preceding input data required by the minimum calculation unit of the last layer network layer, to obtain the preceding output data of the last layer network layer.
[0047] In step S42, the embodiment of the application selects the preceding output data required by the minimum calculation unit of the subsequent network layer from the cache array as the subsequent input data, and inputs the obtained subsequent input data into the subsequent network layer, comprising: when the preceding output data in the cache array meets the minimum data required by the minimum calculation unit of the subsequent network layer, the preceding output data required by the minimum calculation unit of the subsequent network layer is sequentially selected from the cache array as the subsequent input data according to the output order of the preceding output data of the preceding network layer; the obtained subsequent input data is input into the subsequent network layer, and the preceding output data of the preceding network layer in the cache array is synchronously updated.
[0048] That is, for the subsequent network layer, the first subsequent input data is the first few (meeting the minimum data required by the minimum calculation unit of the subsequent network layer) preceding output data of the preceding network layer according to the output order, and the like, after the obtained subsequent input data is input into the subsequent network layer, the preceding output data of the preceding network layer in the cache array is synchronously deleted.
[0049] S5, when the preceding network layer is the last operation network layer in the storage-computing integrated chip, the output result of the storage-computing integrated chip is determined based on all the preceding output data in the cache array.
[0050] With the continuous convolution of the first layer operation network layer, the last layer operation network layer is continuously output, and after the last layer operation network layer no longer outputs, all the preceding output data of the last layer operation network layer are combined to obtain the output result of the storage-computing integrated chip.
[0051] Because all the operation network layers perform operations, even if only the preceding input data and the subsequent input data are cached, when the operation network layer level of the convolutional neural network is too long, the amount of cached data is too large, and even exceeds the cache threshold, that is, the maximum cache in the sum of the data capacity of the input data and the output data of each operation of the operation network layer in the storage-computing integrated chip when performing operation, in order to reduce the amount of cached data, referring to Figure 3 The flowchart for splitting the operation network layer in the storage-computing integrated chip is shown in the figure, and the embodiment of the application further comprises the following steps:
[0052] S61, the sum of the data capacity of the input data and the output data of each operation of the operation network layer in the storage-computing integrated chip when performing operation is calculated respectively, and the maximum data capacity sum is determined as the cache threshold;
[0053] S62, determine whether the cache sum of the preceding input data and the preceding output data of the minimum computing unit corresponding to all the operation network layers exceeds the cache threshold value, when the cache sum corresponding to the operation network layers exceeds the cache threshold value, split the operation network layers in the storage-computing integrated chip to obtain a plurality of sub-models operating independently in sequence;
[0054] S63, when it is detected that the last operation network layer of the preceding sub-model exists preceding model output data, update the preceding model output data to the cache array, and detect whether the preceding model output data in the cache array meets the minimum data required by the minimum computing unit of the first operation network layer of the subsequent sub-model; the preceding sub-model refers to the sub-model currently performing operation in the target storage-computing integrated chip; the subsequent sub-model refers to the sub-model adjacent to the preceding sub-model after the preceding sub-model in the target storage-computing integrated chip;
[0055] S64, when the preceding model output data in the cache array meets the minimum data required by the minimum computing unit of the first operation network layer of the subsequent sub-model, select the preceding model output data required by the minimum computing unit of the first operation network layer of the subsequent sub-model from the cache array as subsequent model input data, input the obtained subsequent model input data into the first operation network layer of the subsequent sub-model, and perform operation on the subsequent model input data by the first operation network layer of the subsequent sub-model to obtain subsequent model output data.
[0056] The embodiment of the present application cuts a complete convolutional neural network into multiple sub-models which are operated independently in sequence, each sub-model is operated independently in sequence according to the connection order, that is, each operation network layer in the preceding sub-model is operated according to the preceding steps S1-S5; when it is detected that there is preceding model output data in the last operation network layer of the preceding sub-model, the preceding model output data is updated to the cache array, and it is detected whether the preceding model output data in the cache array meets the minimum data required by the minimum calculation unit of the first operation network layer in the subsequent sub-model, when the preceding model output data in the cache array meets the minimum data required by the minimum calculation unit of the first operation network layer in the subsequent sub-model, the preceding model output data of the last operation network layer of the preceding sub-model is used as the subsequent model input data of the next subsequent sub-model, each operation network layer in the subsequent sub-model will operate according to the preceding steps S1-S5, and the preceding model output data is updated in the cache array at the same time, that is, after the preceding model output data is input into the subsequent sub-model, the preceding model output data of the preceding sub-model is deleted. If there are multiple sub-models, the preceding steps are sequentially executed according to the connection order of each sub-model, thereby effectively reducing the cache data amount of the cache array.
[0057] In order to further reduce the cache data amount of the cache array, the output result of each sub-model is the preceding output data of the last operation network layer of the sub-model, and the embodiment of the present application can select the operation network layer with the minimum dimension of output data as the cutting object, specifically, the embodiment of the present application cuts the operation network layer in the compute-in-memory chip to obtain multiple sub-models, including: screening all operation network layers based on the dimension of output data of each operation network layer to determine a target network layer, the target network layer is the operation network layer with the minimum dimension of output data, and the target network layer is not the last operation network layer of the compute-in-memory chip; cutting the operation network layer in the compute-in-memory chip based on the target network layer to form a preceding sub-model with all operation network layers before the target network layer and the target network layer, and form a subsequent sub-model with all operation network layers after the target network layer, wherein the target network layer is the last operation network layer of the preceding sub-model.
[0058] When the previous model output data exists in the target network layer of the previous sub-model, the previous model output data is updated to the cache array, and whether the previous model output data in the cache array meets the minimum data required by the minimum calculation unit of the first operation network layer in the subsequent sub-model is detected, and if the previous model output data meets the minimum data required by the minimum calculation unit of the first operation network layer in the subsequent sub-model, the previous model output data of the target network layer is taken as the previous model input data of the subsequent sub-model.
[0059] When the target network layer is the last operation network layer of the previous sub-model, the output result of the previous sub-model is all previous output data after the operation of the target network layer is completed, and the target network layer is the operation network layer with the smallest dimension of output data in all operation network layers, so that the output result cached in the cache array after the operation process of the previous sub-model is completed is the output data with the smallest dimension in all operation network layers, and the output result of the previous sub-model is deleted synchronously after the previous output data is input into the subsequent sub-model, thereby effectively reducing the cache data amount of the cache array.
[0060] In other embodiments of the present application, the plurality of sub-models of the convolutional neural network in the storage-computation integrated chip can be further divided into two or more divided models according to actual application requirements, that is, any sub-model is further divided into two or more divided models, each divided model is independently executed in sequence according to the connection order, and each operation network layer in the divided model is executed according to the steps S1-S5.
[0061] When the minimum calculation unit of the same operation network layer is operated multiple times, if the same data exists between adjacent two previous input data read, in order to reduce the data carrying amount, referring to Figure 4 FIG. 1 is a flowchart of a data multiplexing process for the previous network layer, and the embodiment of the present application further includes the following steps when the previous input data is input into the previous network layer:
[0062] S71, whether the same data exists between adjacent two previous input data input into the previous network layer is determined according to the data dimension of the previous input data, the corresponding convolution kernel size of the previous network layer, and the moving step length.
[0063] For example, when the dimension size of the previous input data is 3x6 and the size of the convolution kernel is 3x3, if the moving step of the convolution kernel is 1, there is same data between the adjacent two previous input data, and correspondingly, if the moving step of the convolution kernel is 3, there is no same data between the adjacent two previous input data.
[0064] S72, if there is same data, the target convolution weight matrix is determined by data reuse sampling on the previous network layer, and when each of the operation network layers performs multiple convolution operations respectively, the previous input data of each of the network layers is processed by reuse according to the target convolution weight matrix.
[0065] For each network layer when performing functional operation, if there is same data in the cached target amount of input data or target amount of output data, in order to reduce the number of data repeated transfer, the embodiment of the application determines the target convolution weight matrix by data reuse sampling on the previous network layer, including: performing one convolution sampling calculation on the target amount of data by the convolution kernel of the previous network layer to obtain a first convolution result; performing second convolution sampling calculation based on the moving step of the convolution kernel to obtain a second convolution result; and determining the target convolution weight matrix when the same data is reused according to the same data of the first convolution result and the second convolution result.
[0066] In the embodiment of the application, the target convolution weight matrix when the same data is reused is determined according to the same data of the first convolution result and the second convolution result, including: converting the first convolution result into a first input sequence based on the initial convolution weight matrix outputting the first convolution result; converting the second convolution result into a second input sequence based on the initial convolution weight matrix outputting the second convolution result; and determining the target convolution weight matrix according to the sequence position of the same data in the first input sequence and the second input sequence.
[0067] When each of the operation network layers performs multiple convolution operations respectively, the previous input data of each of the network layers is processed by reuse according to the target convolution weight matrix, including: when each of the operation network layers performs multiple convolution operations respectively, determining the same data in each of the convolution results obtained; and processing the convolution results containing the same data by reuse according to the target convolution weight matrix, so that each of the operation network layers only stores the different data in the convolution results when caching.
[0068] In the memory-computing integrated chip, each computing unit in each of the operation network layers stores a set of initial convolution weight matrix (i.e. convolution kernel weight) for functional operation of data, and the embodiment of the application determines the same data and the target convolution weight matrix by data reuse sampling, for example:
[0069] The network layer is a convolution layer, and input data of the convolution layer is shown as follows:
[0070] M0 M1 M2 M5 M6 M7
[0071] A convolution kernel of the convolution layer is shown as follows:
[0072] W0 W1 W5 W6
[0073] The moving step length of the convolution kernel is defined as 1, and when the convolution is performed, a first convolution result of a first convolution sampling calculation is shown as follows:
[0074] M0 M1 M5 M6
[0075] A second convolution result of a second convolution sampling calculation is shown as follows:
[0076] M1 M2 M6 M7
[0077] The initial convolution weight matrix is shown as follows:
[0078]
[0079]
[0080] When the first convolution sampling calculation is performed, the first convolution result is carried, and the first convolution result is output based on the initial convolution weight matrix, and the first convolution result is converted into a first input sequence, and the first input sequence is shown as follows:
[0081] M0 M5 M1 M6
[0082] Correspondingly, when the second convolution sampling calculation is performed, the second convolution result is carried, and the second convolution result is output based on the initial convolution weight matrix, and the second convolution result is converted into a second input sequence, and the second input sequence is shown as follows:
[0083] M1 M6 M2 M7
[0084] Based on the first input sequence and the second input sequence, the sequence position of the same data is determined to be M1 and M6, and the target convolution weight matrix is determined according to the sequence position of the same data in the first input sequence and the second input sequence, for example, the target convolution weight matrix is shown as follows:
[0085] W1 W6 W0 W5
[0086] That is, when transporting the second convolution result, the target convolution weight matrix only needs to transport the different data M2, M7 in the second input sequence to replace the different data M0, M5 in the first input sequence, and adjust the sequence positions of the data M1, M6, M2, M7 in the second input sequence according to the target convolution weight matrix (for example, in the above embodiment, the sequence positions of the data M2, M7 in the second input sequence are adjusted from the 3rd column and the 4th column to the 1st column and the 2nd column, and the sequence positions of the data M1, M6 are changed to the 3rd column and the 4th column), so as to multiplex the convolution result containing the same data according to the target convolution weight matrix, and each operation network layer only stores the different data in the convolution result when buffering.
[0087] Compared with the prior art, the convolutional neural network optimization method for the memory-compute integrated chip has the following advantages:
[0088] 1. The preceding input data required by the minimum calculation unit of each operation network layer in the convolutional neural network is determined respectively, and only the preceding input data required by the minimum calculation unit of each operation network layer is cached when performing operation, which greatly reduces the cache occupation and saves the cost and area size of the memory-compute integrated chip.
[0089] 2. Whether the convolutional neural network needs to be split is determined according to the size relationship between the sum of the preceding input data and the preceding output data of all the minimum calculation units corresponding to the operation network layers and the cache threshold value, and after the preceding output data of the preceding sub-model is read by the latter sub-model, the preceding output data cached by the preceding sub-model is deleted synchronously, thereby avoiding the problem of excessive cache occupation when the number of levels of the convolutional neural network is too large.
[0090] 3. When it is determined that there is same data between the adjacent two preceding input data input by the preceding network layer, a target convolution weight matrix is determined by data multiplexing sampling of each operation network layer, and the data output result cached by each operation network layer is multiplexed according to the target convolution weight matrix, thereby reducing the number of data transportation times and improving the data transportation efficiency.
[0091] On the basis of the above embodiment, as an implementation of the method shown in the above Figure 1 The present application provides an embodiment of a convolutional neural network optimization device for a memory-compute integrated chip, which corresponds to the method embodiment shown in the above Figure 1 The device can be applied to various electronic devices, as shown in the above Figure 5 The convolutional neural network optimization device for a memory-compute integrated chip comprises:
[0092] The data operation module 100 is configured to determine a minimum calculation unit corresponding to each operation network layer and a target operation number respectively, read previous input data required by the minimum calculation unit of a previous network layer from the cache array, input the read previous input data into the previous network layer, write previous output data obtained by performing operation on the previous input data in the previous network layer into the cache array and update an actual operation number of the previous network layer after the previous network layer performs operation on the previous input data to obtain the previous output data, wherein the previous network layer refers to an operation network layer currently performing operation in the target memory-compute integrated chip.
[0093] The data update module 200 is configured to update the previous input data in the cache array and continue to determine previous output data corresponding to the updated previous input data and continue to update the actual operation number of the previous network layer when the actual operation number is less than the target operation number, and determine that the previous network layer completes operation when the actual operation number of the previous network layer is equal to the target operation number of the previous network layer.
[0094] The data output module 300 is configured to detect whether the previous network layer is the last operation network layer of the memory-compute integrated chip, and determine an output result of the memory-compute integrated chip based on all previous output data in the cache array when the previous network layer is the last operation network layer of the memory-compute integrated chip.
[0095] The convolutional neural network optimization device for the memory-compute integrated chip provided in the embodiment of the present application can execute the convolutional neural network optimization method for the memory-compute integrated chip provided in the above embodiment, has the corresponding functional steps and beneficial effects of the convolutional neural network optimization method for the memory-compute integrated chip provided in the above embodiment, and specific details can be referred to the above embodiment of the convolutional neural network optimization method for the memory-compute integrated chip. The present application will not be described here again.
[0096] The electronic device can include a processor and a memory, where the processor and the memory can be connected through a bus or other manners. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions / modules of the method for optimizing a convolutional neural network for a chip integrating computing and calculation according to the embodiments of the present application. The processor performs various function applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the method for optimizing a convolutional neural network for a chip integrating computing and calculation according to the embodiments of the present application.
[0097] The memory can include a program storage area and a data storage area, where the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. The one or more modules are stored in the memory and, when executed by the processor, perform the method for optimizing a convolutional neural network for a chip integrating computing and calculation according to the embodiments of the present application. The above electronic device can correspond to the relevant description and effects of the method embodiments, and will not be described here. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above embodiments. The storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.
[0098] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.
[0099] Similarly, it is to be understood that the embodiments of the present application can be readily combined with one another and / or various aspects of the application can be readily combined with one another, to form further embodiments of the application. Similarly, it is to be understood that the various features described herein can form the subject of separate independent embodiments of the application. In addition, it is to be understood that references to a specific feature or features in the description do not mean that the application is limited to only those embodiments which include the specific feature or features. In other words, the scope of the application is not limited to only those embodiments which include the specific feature or features. In addition, it is to be understood that the use of "or" in the description herein is used either in the sense of an inclusive "or" or in the sense of an exclusive "or". In other words, the use of "or" in the description herein is used in the same sense as it is used in the claim. Furthermore, it is to be understood that the use of "a", "an" and "the" in the description herein are open, that is, they mean "one or more", unless specifically indicated to the contrary.
Claims
1. A convolutional neural network optimization method for in-memory computing chips, applied to in-memory computing chips, wherein the in-memory computing chip includes a cache array and multiple computational network layers, characterized in that, The convolutional neural network optimization method for in-memory computing chips includes: Determine the minimum computational unit and the target number of operations for each of the aforementioned computational network layers; The preceding input data required by the smallest computing unit of the preceding network layer is read from the cache array, and the read preceding input data is input into the preceding network layer. After the preceding network layer performs operations on the preceding input data to obtain preceding output data, the preceding output data is written into the cache array and the actual number of operations of the preceding network layer is updated. The preceding network layer refers to the computing network layer currently performing operations in the target in-memory computing chip. When the actual number of operations is less than the target number of operations, the preceding input data in the cache array is updated, and the preceding output data corresponding to the updated preceding input data is determined, and the actual number of operations of the preceding network layer is updated, until the actual number of operations of the preceding network layer is equal to the target number of operations of the preceding network layer, at which point the preceding network layer is determined to have completed the operation. Detect whether the preceding network layer is the last computing network layer of the in-memory computing chip; When the preceding network layer is the last computational network layer in the in-memory computing chip, the output result of the in-memory computing chip is determined based on all preceding output data in the cache array.
2. The convolutional neural network optimization method for in-memory computing chips according to claim 1, characterized in that, After detecting whether the preceding network layer is the last computational network layer of the in-memory computing chip, the method further includes: When the preceding network layer is not the last computational network layer of the in-memory computing chip, if updated preceding output data is detected in the cache array, it is checked whether the preceding output data in the cache array meets the minimum data amount required by the minimum computing unit of the following network layer; the following network layer refers to the computational network layer that is adjacent to the preceding network layer and follows the preceding network layer. When the output data in the cache array meets the minimum data volume required by the minimum computing unit of the subsequent network layer, the output data required by the minimum computing unit of the subsequent network layer is selected from the cache array as the input data. The obtained input data is input into the subsequent network layer. After the subsequent network layer performs operations on the input data to obtain the output data, the output data is written into the cache array and the actual number of operations in the subsequent network layer is updated. When the actual number of operations in the subsequent network layer is equal to the target number of operations in the subsequent network layer, it is determined that the subsequent network layer has completed the operation. Determine whether the following network layer is the last computing network layer of the in-memory computing chip. If the following network layer is not the last computing network layer of the in-memory computing chip, determine the computing network layer adjacent to the following network layer and after the following network layer as the new following network layer, and use the following network layer as the new preceding network layer to continue the computing process of the new following network layer. Until the new subsequent network layer is the last computing network layer of the in-memory computing chip, the output result of the in-memory computing chip is determined based on the subsequent output data of the new subsequent network layer, which is the last computing network layer.
3. The convolutional neural network optimization method for in-memory computing chips according to claim 2, characterized in that, The step of selecting the preceding output data required by the smallest computational unit of the subsequent network layer from the cache array as the subsequent input data, and inputting the acquired subsequent input data into the subsequent network layer includes: When the output data in the cache array meets the minimum data requirement of the smallest computing unit in the subsequent network layer, the output data required by the smallest computing unit in the subsequent network layer is selected from the cache array as the input data in the subsequent layer according to the output order of the output data in the preceding network layer. The acquired subsequent input data is input into the subsequent network layer, and the preceding output data of the preceding network layer in the cache array is updated synchronously.
4. The convolutional neural network optimization method for in-memory computing chips according to claim 1, characterized in that, The method further includes: Calculate the total data capacity of the input and output data of each computing network layer in the in-memory computing chip when performing a computation, and determine the largest total data capacity as the cache threshold. Determine whether the total cached value of the smallest computing unit corresponding to all the computing network layers exceeds the cache threshold. If the total cached value of the computing network layer exceeds the cache threshold, the computing network layer in the in-memory computing chip is segmented to obtain multiple sub-models that are operated independently in sequence. When the output data of the previous model is detected in the last computational network layer of the previous sub-model, the output data of the previous model is updated in the cache array, and it is checked whether the output data of the previous model in the cache array meets the minimum data amount required by the minimum computing unit of the first computational network layer in the subsequent sub-model; the previous sub-model refers to the sub-model currently performing the operation in the target in-memory computing chip; the subsequent sub-model refers to the sub-model in the target in-memory computing chip that is after the previous sub-model and adjacent to the previous sub-model. When the output data of the preceding model in the cache array meets the minimum data amount required by the minimum computational unit of the first computational network layer in the following sub-model, the output data of the preceding model required by the minimum computational unit of the first computational network layer in the following sub-model is selected from the cache array as the input data of the following model. The obtained input data of the following model is input into the first computational network layer of the following sub-model so that the following model output data is obtained by performing operations on the input data of the following model through the first computational network layer of the following sub-model.
5. The convolutional neural network optimization method for in-memory computing chips according to claim 4, characterized in that, The process of segmenting the computational network layer in the in-memory computing chip to obtain multiple sub-models that operate independently in sequence includes: Based on the output data dimension of each of the computing network layers, all the computing network layers are filtered to determine the target network layer. The target network layer is the computing network layer with the smallest output data dimension, and the target network layer is not the last computing network layer of the in-memory computing chip. Based on the target network layer, the computational network layer in the in-memory computing chip is divided to form a preceding sub-model consisting of all computational network layers before the target network layer and the target network layer, and a following sub-model consisting of all computational network layers after the target network layer, wherein the target network layer is the last computational network layer in the preceding sub-model.
6. The convolutional neural network optimization method for in-memory computing chips according to claim 1, characterized in that, When inputting the read preceding input data into the preceding network layer, the method further includes: Based on the data dimension of the preceding input data and the convolution kernel size and stride of the preceding network layer, determine whether there is any identical data between two adjacent preceding input data inputs to the preceding network layer; If identical data exists, the target convolution weight matrix is determined by data reuse sampling of the preceding network layer. When the preceding network layer performs multiple convolutions, the data output results cached by the preceding network layer are reused according to the target convolution weight matrix.
7. The convolutional neural network optimization method for in-memory computing chips according to claim 6, characterized in that, The step of performing data multiplexing sampling on the previous network layer to determine the target convolutional weight matrix includes: The target data is subjected to a convolution sampling calculation by the convolution kernel in the previous network layer to obtain the first convolution result; A second convolution sampling calculation is performed based on the stride of the convolution kernel to obtain the second convolution result; Based on the same data in the first convolution result and the second convolution result, determine the target convolution weight matrix when reusing the same data.
8. The convolutional neural network optimization method for in-memory computing chips according to claim 7, characterized in that, The step of determining the target convolution weight matrix for reusing the same data based on the same data in the first convolution result and the second convolution result includes: The first convolution result is output based on the initial convolution weight matrix, and the first convolution result is converted into the first input sequence. The second convolution result is output based on the initial convolution weight matrix, and the second convolution result is converted into a second input sequence. The target convolution weight matrix is determined based on the sequence positions of the same data in the first input sequence and the second input sequence.
9. A convolutional neural network optimization device for in-memory computing chips, applied to the method described in any one of claims 1-8, characterized in that, The convolutional neural network optimization device for in-memory computing chips includes: The data processing module is used to determine the minimum computing unit and the target number of operations corresponding to each of the computing network layers; read the preceding input data required by the minimum computing unit of the preceding network layer from the cache array, input the read preceding input data into the preceding network layer, perform operations on the preceding input data in the preceding network layer to obtain preceding output data, write the preceding output data into the cache array, and update the actual number of operations of the preceding network layer; the preceding network layer refers to the computing network layer currently performing operations in the target in-memory computing chip; The data update module is used to update the preceding input data in the cache array and continue to determine the preceding output data corresponding to the updated preceding input data when the actual number of operations is less than the target number of operations, and continue to update the actual number of operations of the preceding network layer until the actual number of operations of the preceding network layer is equal to the target number of operations of the preceding network layer, and then determine that the preceding network layer has completed the operation. The data output module is used to detect whether the preceding network layer is the last computing network layer of the in-memory computing chip; when the preceding network layer is the last computing network layer in the in-memory computing chip, the output result of the in-memory computing chip is determined based on all preceding output data in the cache array.
10. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements the instructions of the method as described in any one of claims 1-8.