Method for managing convolution calculations and corresponding device

Abstract

This disclosure relates to methods and corresponding apparatus for managing convolutional computations. In one embodiment, a method for managing convolutional computations performed by a computing unit adapted to compute output data on an output channel based on convolutional kernels applied to input data blocks on at least one input channel, wherein computation for each input data block corresponds to output data on an output channel, and wherein computation for each convolutional kernel corresponds to output data on each output channel, the method comprising: identifying the size of a memory location available in a temporary working memory of the computing unit; preloading a maximum number of convolutional kernels that can be stored within the memory size in the temporary working memory; and controlling the computing unit to compute a set of output data that can be computed based on the preloaded convolutional kernels.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of French patent application No. 2010063, filed on October 1, 2020, which is incorporated herein by reference. Technical Field

[0003] The implementations and examples relate to artificial neural networks that operate convolutional computations, and in particular the management of convolutional computations. Background Technology

[0004] One example that can be mentioned is the convolutional neural network or CNN, which is commonly used to identify objects or people in images or videos; this is called 2D convolution.

[0005] Convolutional neural networks typically contain four types of layers that process information sequentially: convolutional layers, which process one after another, such as image patches; activation layers, which are usually non-linear functions that can improve the relevance of the results from convolutional layers; pooling layers, which can reduce the dimensionality of layers; and fully connected layers (or dense layers), which connect all neurons in one layer to all neurons in the previous layer.

[0006] For each layer, input data arrives at the input channel from the previous layer, and output data is transmitted on the output channel. The input and output channels correspond to memory locations, such as random access memory. The set of output channels is called a "feature map".

[0007] A convolutional layer typically corresponds to the inner product of the input data and the kernel weights. The weights refer to the parameters of the convolution operation associated with a given kernel.

[0008] In short, the principle of convolution (especially 2D convolution) is to scan the input channels using a window projected onto the input data blocks and compute the inner product of each input data block with the convolution kernel. The inner product of each input data block corresponds to the output data on the output channel corresponding to each convolution kernel.

[0009] It should be noted that the convolution kernel corresponds to the output channel and can include multiple components equal to the number of input channels, so each component can be dedicated to one input channel. Therefore, the "absolute" size of such a convolution kernel (i.e., the number of kernel weight data) will be the size of one component (equal to the window size (e.g., 3x3 pixels)) multiplied by the number of input channels. The weights are typically stored in non-volatile memory.

[0010] Convolutional Neural Networks (CNNs) are widely used in artificial intelligence. The computational demands of CNNs are extremely high, requiring substantial non-volatile memory for storing weights, volatile memory for input / output data, and computation cycles that result in high inference times (inference time is a common term in artificial intelligence, representing the time required to perform a task that the neural network has been configured for or trained within a self-learning framework).

[0011] There is a need to reduce the inference time of computations performed in convolutional neural networks.

[0012] The drawback of existing solutions is that they typically do not offer a favorable trade-off between reducing inference time and reducing memory footprint.

[0013] In practice, techniques such as "TVM" generate specific code for each layer to avoid loop execution, which can reduce the number of loops but has a significant impact on the size of non-volatile memory. Furthermore, techniques known as "weight plateauing" or "characteristic plateauing" involve allocating more volatile memory to accelerate computation.

[0014] However, the size of both non-volatile and volatile memory is directly related to the final cost of the solution. In the context of inexpensive devices with limited (random access and non-volatile) memory and computational speed resources, price issues and the trade-off between inference time and memory space have a greater impact. Summary of the Invention

[0015] The embodiments significantly reduce inference time for 2D convolutions. Further embodiments provide disturbance time close to the theoretical limit of the number of loops per convolution computation operation without increasing volatile or non-volatile memory.

[0016] According to one embodiment, a method for managing convolutional computation is provided, wherein the method is performed by a computation unit adapted to compute output data on an output channel based on convolutional kernels applied to input data blocks on at least one input channel, wherein the computation for each input data block corresponds to each output data on the output channel, and the computation for each convolutional kernel corresponds to the output data on each output channel.

[0017] According to one embodiment, the method includes:

[0018] The size of the available memory location in the temporary working memory of the computing unit;

[0019] Preload the maximum number of convolutional kernels that can be stored at this size into the temporary working memory; and

[0020] The control computing unit calculates a set of output data that can be computed based on preloaded convolution kernels.

[0021] Temporary working memory (often called "temporary memory") of a computing unit is a memory location whose intended lifespan is temporary, and it is always provided by any type of code generator or compiler. The size of the available memory location depends on each application and is not always predictable.

[0022] The method according to this embodiment provides optimal use of temporary working memory on a case-by-case basis by preloading the maximum convolutional kernel in it according to the identified available size, which reduces the number of processes for restoring and loading convolutional kernels used in computation.

[0023] It should be noted that this method provides optimal use of available temporary working memory; that is, it does not provide pre-fixed memory allocation.

[0024] Furthermore, the convolution computation performed by the computation unit is controlled after the kernels are preloaded, and these preloaded kernels are used to control the set of output data generated by the convolution. In other words, each preloaded convolution kernel is fully utilized during computation, which again reduces the number of processes for restoring and loading convolution kernels.

[0025] The combination of these two mechanisms, used to reduce the number of processes implemented for recovering and loading convolutional kernels, results in considerable time savings. Firstly, simulations of neural network topologies with 2D convolutional layers show a 98% reduction in total time dedicated to data loading compared to conventional methods, and a 30% to 50% gain in global inference time for the complete neural network. Of course, performance gains will depend on the network topology and implementation conditions.

[0026] Finally, this method is general and can be applied to any organization of input / output data and convolution kernel data, with particular optimization for memory usage, such as organizations that utilize segmented memory storage.

[0027] According to one embodiment, the control of a computing unit for computing a set of output data (which can be computed based on preloaded convolutional kernels) includes: continuously loading input data blocks corresponding to the output data of the set into the computing unit.

[0028] In other words, this management method provides a scan of all input data blocks that can be used with the preloaded cores in order to calculate the maximum output data based on the loaded cores.

[0029] In one implementation, the method includes repeating preloading and control steps until the set of output data in all output channels is calculated.

[0030] In fact, if all convolutional kernels cannot be loaded into the temporary working memory, the process is repeated for the other convolutional kernels after calculating the maximum output data of the output channels corresponding to the preloaded kernels, so as to calculate the maximum output data of the output channels corresponding to these other convolutional kernels.

[0031] Therefore, the input data block is loaded into the processing unit more times than in conventional methods, but this loss can be largely compensated for by the gain in the number of times the convolution kernel is recovered and loaded compared to conventional methods.

[0032] According to one embodiment, the set of output data that can be calculated corresponds to the maximum size of the output data that can be received at one time in each output channel, and the set can include the complete output data of each output channel, or only a portion of the output data of each output channel.

[0033] For example, a portion of the output data for each output channel corresponds to a single line or a set of lines for each output channel.

[0034] In practice, the technique used to recover output channels from input channels can optimize the memory storing input and output data. In this case, the maximum size of output data that can be received at one time in each output channel is limited to a portion, for example, depending on the organization of the memory recovery, limited to one line or a group of lines, so as not to corrupt the input data by overwriting the input data with output data.

[0035] According to one embodiment, if the set only includes a portion of the output data for each output channel, the repetition of the preloading step and the control step includes:

[0036] a) For the same portion, repeat the preloading step with convolution kernels different from those in the previous preloading step and repeat the control step until all output data for the portion has been computed for all output channels, and, in addition,

[0037] b) For the other parts, restart step a) until the output data is calculated for all parts of all output channels.

[0038] In other words, in this embodiment, output data is calculated to first fill portions of all output channels, such as rows. To reach all output channels, the preloading step is repeated using the corresponding output kernels to successively cover portions of all output channels. Then, the process is restarted for the next portion (e.g., the next row) of all output channels.

[0039] Although the step of preloading the convolution kernel is repeated for each part, the method according to this implementation provides a time saving of 89.7% to achieve a reduction in time dedicated to data loading compared to conventional methods.

[0040] According to one embodiment, if the size of the available memory location in the temporary working memory is less than a minimum size threshold, the identification step further includes the allocation of a memory location in the computing unit that is a buffer memory of the minimum size threshold, and preloading is performed in the allocated memory location, having a maximum number of convolutional kernels that can be stored under the minimum size threshold.

[0041] In this embodiment, memory locations are allocated, and therefore the memory size occupied is slightly increased compared to one of the implementations defined above. However, this increase in memory footprint (which can be equal to the size of two convolutional kernels) does not introduce excessive additional costs, especially regarding the gain gained in computation time.

[0042] According to one embodiment, the convolution kernel includes weight data, and the preloading step includes reorganizing the weight data of the convolution kernel to optimize the computation of the output data by the computing unit.

[0043] In other words, regardless of the storage mapping of the convolution kernel, the preloading of the weight data is arranged to reduce inference time.

[0044] It should be noted that input data can also be reorganized in this respect during the sequential loading of input data blocks.

[0045] According to one embodiment, the convolutional kernel includes weight data, and the output data computation includes multiplying and summing the input data of the loaded block with the weight data of the preloaded convolutional kernel.

[0046] According to one embodiment, the convolution kernel is stored in non-volatile memory inside or outside the computing unit before being preloaded in temporary working memory, and input and output data are stored in volatile memory inside or outside the computing unit during the control of the computing unit.

[0047] In fact, even with non-volatile memory within the computing unit, the amount of convolution kernel recovery and loading processes can be reduced to save time by following this approach. However, the gain is even greater with external non-volatile memory. Moreover, even when the input channels are stored in external memory, the resulting time savings still outweigh the time loss caused by the increased number of input data blocks loaded.

[0048] The methods according to these embodiments can be implemented in software or hardware, such as an integrated device that includes, for example, processing components in a state machine logic circuit designed to organize and control the execution of computations.

[0049] The embodiment provides a computer program product including instructions that, when executed by a computer, instruct the computer to implement the methods defined above.

[0050] Another embodiment provides a computer-readable medium including instructions that, when executed by a computer, instruct the computer to implement the methods defined above.

[0051] According to another embodiment, an apparatus is provided, wherein the apparatus (e.g., an integrated apparatus) includes a processing unit configured to manage convolutional computations performed by a computing unit adapted to compute output data on an output channel based on convolutional kernels applied to input data blocks on at least one input channel, wherein the computation for each input data block corresponds to each output data on the output channel, and the computation for each convolutional kernel corresponds to the output data on each output channel, characterized in that: the processing unit is further configured to:

[0052] The size of the available memory location in the temporary working memory of the computing unit;

[0053] Preload the maximum number of convolutional kernels that can be stored at this size into the temporary working memory; and

[0054] The control computing unit calculates a set of output data that can be computed based on preloaded convolution kernels.

[0055] According to one embodiment, the processing unit is configured to control the computing unit to compute a set of output data by sequentially loading input data blocks corresponding to the output data in the set, the set of output data being computed based on preloaded convolutional kernels.

[0056] According to one embodiment, the processing unit is configured to repeatedly preload and control until the set of output data in all output channels is calculated.

[0057] According to one embodiment, the set of output data corresponds to the maximum size of output data that can be received at one time in each output channel. This set can include the complete output data of each output channel, or only a portion of the output data of each output channel.

[0058] According to one embodiment, a portion of the output data for each output channel corresponds to one line or a set of lines for each output channel.

[0059] According to one embodiment, if the set only includes a portion of the output data for each output channel, then for repeated preloading and control, the processing unit is configured to:

[0060] a) For the same portion, repeat the preloading of convolutional kernels different from the previously preloaded kernels and repeat the control until all output data for the portion is computed for all output channels; and, additionally,

[0061] b) For the other parts, start a) again until the output data is calculated for all parts of all output channels.

[0062] According to one embodiment, if the size of the available memory location in the temporary working memory is less than a minimum size threshold, the processing unit is configured to allocate a memory location in the computing unit to a buffer memory of the minimum size threshold, and preload the allocated memory location with the maximum number of convolution kernels that can be stored under the minimum size threshold.

[0063] According to one embodiment, the convolution kernel includes weight data, and the processing unit is configured to: reorganize the weight data of the convolution kernel during preloading, and optimize the computation of the output data by the computing unit.

[0064] According to one embodiment, the convolutional kernel includes weight data, and the computation unit is configured to compute output data by multiplying and summing the input data of the loaded block with the weight data of the preloaded convolutional kernel.

[0065] According to other embodiments, a system is provided, wherein the system includes the device as defined above, a computing unit, a non-volatile memory inside or outside the computing unit for storing convolutional kernels, and a volatile memory inside or outside the computing unit for containing input data and output data. Attached Figure Description

[0066] Other advantages and features of this disclosure will become clear after examining the detailed description of the implementations and embodiments, as well as the accompanying drawings, in which:

[0067] Figure 1 The management process for convolution computation of a neural network according to the first embodiment is illustrated;

[0068] Figure 2 The management process for convolution computation of a neural network according to the second embodiment is illustrated;

[0069] Figure 3 An implementation of the second embodiment is shown in the case where I / O is restored to negative.

[0070] Figure 4An implementation of the second embodiment is shown in the case where I / O is restored to a positive state;

[0071] Figure 5 An implementation of a conventional method for managing convolution computations with the same input data and the same convolution kernel to achieve the same output data is shown for comparison; and

[0072] Figure 6 An exemplary system adapted to implement the method according to the embodiments is shown. Detailed Implementation

[0073] Figure 1 The management process 100 for convolution computation of a neural network of, for example, a 2D convolution type is shown.

[0074] The mathematical operations for convolution are performed by the computational unit UC( Figure 6 The computation unit UC is adapted to apply a convolution kernel to a block of input data in at least one input channel in order to compute output data on the output channel.

[0075] For example, in the case where the input data is pixels of an "RGB" (red, green, blue) image, the input data is provided on input channels for the red "R" component, the green "G" component, and the blue "B" component. A convolutional kernel consists of a matrix of weight data (e.g., 3x3 weight data) the size of a convolutional window. Inner products (i.e., multiplication and summation) are computed between the pixels of the input data block, which is the size of the convolutional window, for each input channel and the kernel weight data. Typically, but not necessarily, a convolutional kernel includes a weight data matrix dedicated to each input channel. The sum of the inner products obtained on each input channel constitutes the output data at the position in the output channel corresponding to the convolutional window. The same operation on another convolutional kernel provides output data at the same position in another output channel. The convolutional window scans all of each input channel to provide all output data, also known as a "feature map." The output channels of the feature map can then be used as input channels for the next layer of the neural network.

[0076] In summary, computation on the input data block produces output data on the output channel, and computation using the convolution kernel produces output data on the output channel.

[0077] The computation performed on each of the convolutional kernels across all input data blocks (scanned on the input channels) thus produces complete output data across all output channels, called a "feature map".

[0078] Management method 100 controls and organizes the process of convolution computation, particularly the selection and access of input data and weight data used in the computation. Weight data is typically stored in non-volatile memory, while input and output data are typically stored in volatile memory.

[0079] In the initial stages of computation, the computation unit is programmed to execute instructions that implement convolution computation, and the temporary working memory of the computation unit is provided by any code generator or compiler.

[0080] The temporary working memory (often called "temporary memory") of a computing unit is a memory location whose intended use is temporary; for example, it typically allows temporary storage of intermediate data used to perform computational operations. The size of the available memory location depends on each application and is not always predictable, but it can, however, be known at the output of the code generator.

[0081] The method includes an identifier 110 for the size of the memory location available in the temporary working memory Temp.Work.Mem of the computing unit.

[0082] During this first identification phase, the method may also include initializing the parameter ch_out_remaining, which represents the number of remaining output channels to be computed. This parameter is initialized to the total number of output channels of the running convolutional layer, nb_ch_out, which is the number of convolutional kernels used by that layer.

[0083] Advantageously, the method can provide test 112, which checks whether the size of the available memory location in the temporary memory Temp.Work.Mem is greater than the minimum size threshold size_min.

[0084] The minimum size threshold size_min is advantageously equal to the size of the two convolution kernels.

[0085] In the case of affirmative A in test 112 (i.e., the size of the available space in the temporary working memory is greater than the minimum size threshold size_min), the maximum number of convolutional kernels nb_max that can be stored in the available space of the temporary working memory is calculated as 120.

[0086] The calculation of 120 can be performed based on the size and dimensions of the identified convolutional kernel, for example: nb_max = PE(size / Kx*Ky*nb_ch_in), where "PE()" is an integer function, "size" is the identified size, "Kx" and "Ky" are the horizontal and vertical dimensions of the convolutional kernel weight data matrix, and "nb_ch_in" is the number of input channels.

[0087] In some cases (which may be the case most of the time when the dimensions Kx, Ky and the number nb_ch_in are large enough), the available memory space is insufficient.

[0088] In this regard, in the case of the negation N of test 112 (i.e., the size of the available memory location in the temporary working memory is less than the minimum size threshold size_min), the memory location of the buffer memory of the computing unit is allocated 122.

[0089] The size of the allocated buffer location is advantageously equal to the size of the minimum size threshold, size_min. In this case, the value nb_max is set to size_min.

[0090] In the following text, the location allocated in the buffer memory will be considered to constitute the temporary working memory Temp.Work.Mem.

[0091] The method then includes preloading 130 of the maximum number of convolutional kernels nb_max that can be stored in the temporary working memory Temp.Work.Mem.

[0092] If the available size in the Temp.Work.Mem is greater than all remaining convolution kernels ch_out_remaining, then only the remaining number of convolution kernels are preloaded.

[0093] Once the convolution kernel is preloaded in the temporary working memory of the computation unit, the convolution computation is controlled 140.

[0094] In this regard, the preloading step 130 of method 100 can advantageously implement techniques for reorganizing the weight data of the convolution kernel, thereby optimizing the computation of the output data by the computation unit 140.

[0095] The convolution computation performed by the computing unit is controlled for all possible output data, which are generated by convolution using a preloaded kernel.

[0096] Therefore, the convolution computation control 140 includes sequential (146-N) loading 142 of the input data block block_Dat_E in the computation unit. For each loading 142, the computation unit performs a series of multiplications and accumulated MAC (inner product) of the input data of the loaded block_Dat_E with the weight data of the convolution kernel Ker_record preloaded in step 130 to provide the output data Dat_S.

[0097] As long as the set of output data Dat_S that can be computed based on the preloaded convolution kernel is not completed (146-N), the next input data block is loaded 142 and the computation of the corresponding inner product Dat_S is performed 146.

[0098] When the set of output data Dat_S that can be computed based on the preloaded convolution kernel is complete (146-A), the method includes repeating the preloading step 130 and the convolution computation control step 140 until all output data in all output channels has been computed 160.

[0099] In this regard, test 150 is provided to check if there are any remaining output channels ch_out_remaining to be calculated.

[0100] In the case of a positive result in test 150, the remaining number of output channels ch_out_remaining is updated in 152, and the preloading step 130 and the convolution calculation control step 140 are repeated, where the maximum number of convolution kernels nb_max corresponds to the remaining output channels.

[0101] Furthermore, if the number of remaining convolutional kernels ch_out_remaining is less than the maximum number nb_max, then only the remaining number of convolutional kernels is preloaded.

[0102] In the case of a negative N of test 150, the set of output data DatS that can be calculated is obtained in all output channels 160.

[0103] The set of output data that can be computed is limited by the maximum size of the output data that can be received at one time in each output channel.

[0104] Therefore, the set of output data that can be calculated corresponds to the maximum size of output data that can be received at one time in each output channel.

[0105] The maximum size of output data that can be received at one time in each output channel is often limited by the amount of memory used to store both input and output data. In fact, there are techniques that allow optimizing this memory size by overlapping or interleaving input and output data. In this type of technique, not all output data can be written to memory without overwriting (or destroying) input data that should still be used for subsequent calculations.

[0106] On the other hand, if the memory organization allows, for example, if there is a memory location dedicated to output data that is different from the input data, then the size of the output data that can be received at one time in each channel can contain the complete output data of the output channel.

[0107] The set of output data that can be computed can therefore include the complete output data for each output channel, or only a portion of the output data from each output channel. A portion can, for example, correspond to a single line or a set of lines of output data.

[0108] Figure 2 As previously shown regarding Figure 1 The described convolution computation management method 200 distinguishes between the two possibilities mentioned above and is related to the set of output data that can be computed. Figure 1 The related steps have the same reference numerals and will not be described in detail hereafter.

[0109] In particular, preparatory steps 110, 112, 120, and 122 are strictly identical.

[0110] At the end of preparatory steps 110 to 122, test 210 is provided to check, for example, whether input and output I / O data are recorded using overlapping or interleaving techniques.

[0111] Test 210 is more broadly designed to identify whether the maximum size of output data that can be received at one time in each output channel corresponds to a portion of the output channel.

[0112] In the case of the negation N of test 210 (i.e., the maximum size of the output data that can be received at one time in each output channel corresponds to the complete output channel), the steps of preloading 130, calculation control 140, and their repetitions 150 and 152 are implemented in branch 300, with regard to Figure 1 The way they are described is not particularly different.

[0113] The set of output data Dat_S acquired in each computation cycle controlled in step 140 therefore corresponds to all output data of each integer output channel / ch_out corresponding to the preloaded convolution kernel.

[0114] In other words, the computation for the nb_max integer output channel was performed in a single stage with a preloaded convolution kernel of 130.

[0115] Then, the convolution kernels corresponding to the remaining output channels (150 to 152) to be computed are preloaded, and the full feature map MAP(ch_out) is obtained in 270.

[0116] In the case of affirmative A in test 210 (i.e., if the maximum size of the output data that can be received at one time in each output channel corresponds to the output channel portion), the steps of preloading 130, calculation control 140, and their repetitions 150 and 152 are related to... Figure 1The method described is implemented in a similar way in branch 400, but in addition, conditional repetition 260 is performed on the remainder of the output channel.

[0117] In fact, the repetition of the preloading step 130 for the same part with a convolution kernel different from the convolution kernel of the previous preloading step 130, and the repetition of the calculation control step 140, result in the calculation of all output data for that part in all output channels.

[0118] Therefore, in order to calculate the output data belonging to the subsequent part, the method includes, on the one hand, test 260, which is provided to test whether there is any part left to be calculated.

[0119] In the case of a positive result in test 260, on the one hand, the parameter ch_out_remaining is reset to its initial value, i.e., ch_out_remaining = nb_ch_out, and the output channel section is updated during step 262.

[0120] On the other hand, the steps of preloading 130, calculating control 140 and repeating them 150, 152 are restarted, but for subsequent parts of the output channel, until the output data is calculated in all parts of all output channels.

[0121] It should be noted that the output channel portion corresponds to a set of output data located in the output channel, typically one row or a set of rows in the output channel. Therefore, restarting the preloading step 130 and the computation control step 140 for the other portion corresponds to the normal preloading of the convolution kernel, and to controlling the computation by loading other input data blocks corresponding to the positions of the output data belonging to the other portion.

[0122] Therefore, when all output channel portions of all output channels are calculated, the total feature map MAP(ch_out) is obtained as 270.

[0123] Now for reference Figures 3 to 5 .

[0124] Figure 3 This schematically illustrates the case 300 regarding the negation N in test 210. Figure 2 The implementation of the described method 200.

[0125] Figure 4 This schematically illustrates the situation in case 400 of affirmative A in test 210. Figure 2 The implementation of the described method 200.

[0126] Figure 5An implementation of a conventional method 500 for managing convolutional computations with the same input data and the same convolutional kernel to achieve the same output data is illustrated schematically for comparison.

[0127] exist Figure 3 , Figure 4 and Figure 5 In this array, the six output channels ch_out_1 to ch_out_6 are represented as an output data matrix, with each output data represented by a box containing numbers and letters.

[0128] Recall that each output data is generated by computation between the input data block loaded in the computation unit and the convolution kernel preloaded in the temporary working memory of the computation unit.

[0129] In each box, the number changes (increments numerically) each time the convolution kernel used to compute the corresponding output data must be preloaded; and the letter changes (increments alphabetically) each time the input data block used to compute the corresponding output data must be loaded.

[0130] The use of Latin and Greek letters is a simple symbolic approach. For convenience, the first letters AE and α-ε are used to indicate the first load of the input block, and the last letters Z and ω are used to indicate the last load of the input block. It is by no means intended to limit the total number of loads by the number of letters that make up the corresponding alphabet.

[0131] It should be noted that Figures 3 to 5 The example in the text corresponds to a convolutional computation technique commonly known as "dual MAC" implemented by a computational unit. Dual MAC is well known to those skilled in the art and simply allows, for example, the simultaneous computation of two output data on two output channels based on the adapted input data block and based on two convolutional kernels via a single multiplication and accumulation mechanism.

[0132] Figure 3 and Figure 4 This corresponds to the case where the available memory location in the temporary working memory of the computing unit can contain up to five convolution kernels (nb_max = 5).

[0133] Therefore, in Figure 3 In the example, the first preloading step "1" allows the first five convolutional kernels to be stored in temporary working memory.

[0134] Scan the set of input data blocks “A”, “B”, “C”, “D”, “E”, …, “Y”, “Z”, and combine them with the preloaded convolutional kernels “1A”, …, “1Z” (MAC) to obtain the complete set of output data MAP corresponding to the five output channels ch_out_1, ch_out_2, ch_out_3, ch_out_4, and ch_out_5 of the five preloaded convolutional kernels.

[0135] Then, with only the last convolution kernel remaining, the second preloading step "2" allows the convolution kernel corresponding to the sixth output channel ch_out_6 to be stored in temporary working memory.

[0136] The set of input data blocks “α”, “β”, “γ”, “δ”, “ε”, …, “ω” is scanned again and combined with the preloaded convolution kernels “2α”, …, “2ω” (MAC) to obtain the complete set of output data MAP of the sixth output channel ch_out_6 corresponding to the preloaded convolution kernels.

[0137] exist Figure 4 In the example, the first preloading step "1" allows the first five convolutional kernels to be stored in temporary working memory.

[0138] Scan the input data block (e.g., the first row of STRP) “A”, “B”, “C”, “D” corresponding to the first part of the output channel, and combine it with the preloaded convolution kernels “1A”...“1D” (MAC) to obtain the first row of STRP of the output data of the five output channels ch_out_1, ch_out_2, ch_out_3, ch_out_4, ch_out_5 corresponding to the five preloaded convolution kernels.

[0139] The second preloading step “2” allows the convolution kernel corresponding to the sixth output channel ch_out_6 to be stored in temporary working memory.

[0140] Scan the input data blocks “α”, “β”, “γ”, “δ” corresponding to the output channel portion, and combine them with the preloaded convolution kernels “2α”...“2δ” (MAC) to obtain the first row of the output data of the sixth output channel ch_out_6 corresponding to the preloaded convolution kernel.

[0141] The third preloading step “3” allows the first five convolutional kernels to be stored in temporary working memory, and scans the input data block “E”... corresponding to the subsequent part, and combines it with the preloaded convolutional kernel “3E”... (MAC) to obtain the second row of STRP of the output data of the five output channels ch_out_1, ch_out_2, ch_out_3, ch_out_4, ch_out_5 corresponding to the preloaded convolutional kernels.

[0142] The fourth preloading step “4” allows the convolution kernel corresponding to the sixth output channel ch_out_6 to be stored in temporary working memory, and scans the input data block “ε”... corresponding to the subsequent part, and combines it with the preloaded convolution kernel “4ε”... (MAC) to obtain the second row of STRP of the output data of the sixth output channel ch_out_6 corresponding to the preloaded convolution kernel.

[0143] The steps of preloading and scanning the input data blocks corresponding to the subsequent sections are restarted until the output data is calculated in all sections of all output channels..., "13Y", "13Z", ..., "14ω".

[0144] exist Figure 5 In the example, conventional method 500 first loads the input data block "A", and then loads three consecutive kernel pairs "1", "2", and "3" for the input data block "A" corresponding to the output data "1A", "2A", and "3A" on the first and second output channels ch_out_1, ch_out_2, the third and fourth output channels ch_out_3, ch_out_4, and the fifth and sixth output channels ch_out_5, ch_out_6, respectively.

[0145] Then, the conventional method first loads the next input data block "B", and then, for the input data block "B", loads three consecutive convolution kernel pairs "4", "5", and "6" corresponding to the output data "4B", "5B", and "6B" on the corresponding output channel pairs.

[0146] For each input data block loaded during the input channel scan, the three loading processes of the convolution kernel are repeated.

[0147] It should be noted that, due to the dual-MAC technique described above, convolutional kernel loading is performed in pairs, and due to the total number of six kernels, there are three pairs for each input data block. Without the dual-MAC technique, there are typically six consecutive loadings of a single convolutional kernel for each input data block.

[0148] Therefore, in this arbitrary example, the conventional approach uses ninety-six (96) loads of six convolutional kernels and the full AZ load of the alphabet, for example, fifty-six (56) loads of the input data block.

[0149] In contrast, in the same arbitrary example, method 300 uses two (2) preloads of six convolutional kernels and two equivalent alphabets, i.e., one hundred and twelve (112) loads of the input data block.

[0150] In contrast, in the same arbitrary example, method 400 uses fourteen (14) preloads of six convolutional kernels and two equivalent alphabets, i.e., one hundred and twelve (112) loads of the input data block.

[0151] Table 1 below shows the results of conventional method 500, indicated in bold, as well as the advantageous methods 300 and 400 implemented above.

[0152] The rows in Table 1 correspond to the following elements:

[0153] nb_ch_in: The number of input channels;

[0154] Kx: The number of weight data columns in the convolution kernel;

[0155] Ky: The number of rows of weight data for the convolution kernel;

[0156] nb_max: The maximum number of convolutional kernels that can be stored within the size of the available memory locations in the temporary working memory of the computation unit;

[0157] nb_ch_out: The number of output channels;

[0158] out_x: The number of columns of output data for the output channel;

[0159] out_y: The number of rows of output data for the output channel;

[0160] LA: Loading time of the input data block, arbitrarily expressed in loop counts (c);

[0161] LW: Kernel loading time, arbitrarily expressed in loop counts (c);

[0162] 500: The total cumulative loading time of the input data block and convolution kernel for the conventional method 500, arbitrarily in millions of cycles (Mc);

[0163] 300: Total cumulative loading time of input data blocks and convolutional kernels for method 300, arbitrarily in millions of cycles (Mc);

[0164] 400: Total cumulative loading time of the input data block and convolution kernel for method 400, arbitrarily in millions of cycles (Mc).

[0165] It should be noted that, under all cases C1-C7 in Table 1, except for case C3, methods 300 and 400 provide significantly less cumulative total time for loading input data blocks and convolutional kernels than the conventional method 500.

[0166] In particular, attention should be paid to case C7, for which methods 300 and 400 offer time savings of 98% and 89.7% respectively compared to the conventional method 500. Case C7 is characterized by the relatively large numbers nb_max (nb_max = 30) and LW >> LA (3 >> 1).

[0167] Furthermore, it should be noted that method 300 is essentially equivalent to the conventional method 500, and method 400 is slightly slower (+8%) than the conventional method 500. The characteristic of C3 lies in the minimum number of operations, nb_max (nb_max = 2), and LW. <LA(2<1)。

[0168] Case C3 actually corresponds to the theoretical suboptimal of the computational conditions. In practice, the input data is typically stored in volatile memory, while the convolution kernel weights are typically stored in non-volatile memory. However, accessing data in volatile memory LA is usually, and in most cases, faster than accessing data in non-volatile memory LW (and therefore, generally, LW > LA).

[0169] [Table 1]

[0170]

[0171]

[0172] Figure 6 This shows that it is adapted to implement about Figures 1 to 4 An example of the system SYS for the described method.

[0173] The system SYS includes devices including a processing unit ME (also known as a "state machine") and a computing unit UC, which includes internal non-volatile memory NVM_I and internal volatile memory RAM_I. The system SYS also includes external non-volatile memory NVM_E and external volatile memory RAM_E.

[0174] The state machine ME and the computing unit UC can be implemented, for example, as devices integrated into integrated circuits or integrated into a system-on-a-chip.

[0175] External storage devices NVM_E and RAM_E are connected to the computing unit UC via a communication interface, such as a BUS or IUS of "SPI" (an acronym for "Serial Peripheral Interface" known to those skilled in the art). 2 The C (an acronym for “internal integrated circuit” as known to those skilled in the art) type, the AMBA (an acronym for “Advanced Microcontroller Bus Architecture” as known to those skilled in the art) series of buses, as part of a “System-on-Chip” type system or any other compatible type of bus.

[0176] External non-volatile memory NVM_E is, for example, EEPROM memory or "flash memory", such as "flash QSPI memory" for SPI type buses, or any other type of adapter memory.

[0177] The external volatile memory RAM_E is, for example, random access memory of the type “SDRAM” (an acronym for “Synchronous Dynamic Random Access Memory” known to those skilled in the art), or any other type of adaptable memory.

[0178] Internal non-volatile memory NVM_I is, for example, EEPROM memory or "flash memory" or any other type of adapter memory integrated into the computing unit UC.

[0179] The internal volatile memory RAM_I is, for example, random access memory of the "AXIRAM" type integrated into the computing unit UC for a microcontroller bus of the AXI (an acronym for "Advanced Extensible Interface" known to those skilled in the art), or any other type of adaptable memory.

[0180] When performing convolution calculations, the convolution kernel weight data is stored in internal non-volatile memory NVM_I or external non-volatile memory NVM_E; while the input and output data are contained in internal volatile memory RAM_I or external volatile memory RAM_E.

[0181] The computation unit UC is adapted to perform convolution computation as defined above, that is, in specific operations of multiplication and accumulation between input data and weight data to produce output data feature cards.

[0182] The processing unit of the state machine ME is configured according to the above description. Figures 1 to 4 The described convolution computation management method manages the computations performed by the computation unit UC.

[0183] Specifically, the processing unit of the state machine ME is configured to identify 110 ( Figure 1 and 2The size of the available memory location in the temporary working memory of the computing unit; preload 130 ( Figure 1 and 2 The maximum number of convolutional kernels that can be stored at this size; and control 140 ( Figure 1 and 2 The computational unit computes the set of output data that can be computed based on the preloaded convolutional kernel.

[0184] In this respect, the state machine ME can generate signals to control the computational unit UC, in particular to identify the size of the available memory location in the temporary working memory; and can generate memory access control, in particular to select and access the input data block to be loaded, and the convolution kernel to be preloaded.

[0185] In this respect, the processing unit of the state machine ME can be "hard-configured," that is, for example, Figure 2 As shown, and advantageously in an optimized and accelerated manner, the logic circuit for implementing the decision tree is constructed.

[0186] The processing unit can also be software configured in this respect by a program, for example, recorded on a computer medium, including instructions that drive the state machine ME to implement the method during program execution. In this case, the processing unit is typically integrated into a microcontroller or microprocessor integrated circuit, which may also incorporate a computing unit UC.

[0187] Due to the optimal use of temporary working memory, regarding Figures 1 to 4 The described implementation and about Figure 6 The described embodiments greatly optimize the execution of convolutional layers in neural networks, significantly reduce the inference time of neural networks, and come very close to the theoretical limit in terms of the number of loops through multiplication and accumulation operations, without increasing volatile memory RAM_I, RAM_E and non-volatile memory NVM_I, NVM_E.

[0188] The advantageous results of the implementations and embodiments shown in Table 1 are not particularly conditional on the location of the weight data (i.e., in internal or external non-volatile memory) or the location of the input data (i.e., in internal or external volatile memory).

[0189] The reduction in inference time allows for lower energy consumption, lower cost of microcontrollers, more complex use cases, and always without increasing memory or reducing performance, such as in terms of accuracy.

[0190] Furthermore, the implementation and examples are compatible with any type of topology of convolutional neural networks and are virtually independent of the microprocessor or microcontroller used, and independent of the bit width of the data (e.g., 32-bit floating-point, 8 quantization bits or less).

Claims

1. A method for managing convolutional computation performed by a computing unit in an image recognition system, the computing unit being adapted to compute output data on an output channel based on convolutional kernels applied to input data blocks on at least one input channel, wherein the computation for each input data block corresponds to output data on the output channel, and wherein the computation for each convolutional kernel corresponds to the output data on each output channel, the method comprising: The size of the available memory location in the temporary working memory of the computing unit is identified, wherein the temporary working memory is a temporary memory inside the computing unit used to temporarily store convolution kernels; The maximum number of convolutional kernels that can be stored within the size of the temporary working memory are preloaded in the temporary working memory to optimize the number of kernel loadings. as well as The computing unit is controlled to compute a set of output data, which can be computed based on preloaded convolutional kernels. By preloading the maximum number of convolutional kernels and computing the set of output data, the number of times the convolutional kernels are loaded is reduced, thereby reducing the inference time and memory usage of convolutional computation.

2. The method according to claim 1, wherein controlling the computing unit to calculate the set of output data comprises: The input data blocks corresponding to the set of output data are loaded sequentially in the computing unit.

3. The method of claim 1 further comprises repeatedly preloading the maximum number of convolution kernels in the temporary working memory and repeatedly controlling the computing unit to compute the set of output data until the set of output data in all output channels is computed.

4. The method of claim 1, wherein the set of output data corresponds to the maximum size of the output data that can be received at one time in each output channel, and wherein the set includes the complete output data of each output channel or only a portion of the output data of each output channel.

5. The method of claim 4, wherein the portion of the output data of each output channel corresponds to a line or a group of lines of output data.

6. The method according to claim 4, wherein, When the set includes only a portion of the output data for each output channel, repeated preloading and control include: For the same portion, convolutional kernels different from those previously preloaded are repeatedly preloaded and controlled repeatedly until all output data for the portion is computed for all output channels; and For the other parts, restart the preloading process until all the output data in all other parts of all the output channels have been calculated.

7. The method according to claim 1, further comprising: When the size of the available memory location in the temporary working memory is less than the minimum size threshold, a memory location in the computing unit with a buffer memory of the minimum size threshold is allocated, wherein preloading the maximum number of convolutional kernels includes: preloading the maximum number of convolutional kernels that can be stored under the minimum size threshold.

8. The method of claim 1, wherein the convolution kernel includes weight data, and wherein preloading includes reorganizing the weight data of the convolution kernel to optimize the computation of the output data.

9. The method of claim 1, wherein the convolution kernel includes weight data, and wherein controlling the computing unit to compute the set of output data includes: The loaded input data block is multiplied and accumulated with the weight data of the preloaded convolution kernel.

10. The method according to claim 1, further comprising: Before the convolution kernel is preloaded into the temporary working memory, the convolution kernel is stored in a non-volatile memory inside or outside the computing unit, and the input data and the output data are stored in the volatile memory inside or outside the computing unit while under the control of the computing unit.

11. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, perform the method according to claim 1.

12. A device in an image recognition system, comprising: The computing unit is configured to compute output data on an output channel based on convolution kernels applied to input data blocks on at least one input channel, such that the computation for each input data block corresponds to each output data on the output channel, and the computation for each convolution kernel corresponds to the output data on each output channel. as well as The processor is configured to manage the convolution computations performed by the computing unit. The processor is also configured to: The size of the available memory location in the temporary working memory of the computing unit is identified, wherein the temporary working memory is a temporary memory inside the computing unit used to temporarily store convolution kernels; The maximum number of convolutional kernels that can be stored within the size of the temporary working memory are preloaded in the temporary working memory to optimize the number of kernel loadings. as well as The computing unit is controlled to compute a set of output data, which can be computed based on preloaded convolutional kernels. By preloading the maximum number of convolutional kernels and computing the set of output data, the number of times the convolutional kernels are loaded is reduced, thereby reducing the inference time and memory usage of convolutional computation.

13. The apparatus of claim 12, wherein the processor is configured to: control the computing unit to compute the set of output data by sequentially loading input data blocks corresponding to the set of output data in the computing unit, the set of output data being computed based on the preloaded convolution kernel.

14. The apparatus of claim 12, wherein the processor is configured to: repeatedly preload the maximum number of convolutional kernels in the temporary working memory and repeatedly control the computing unit to compute the set of output data until the set of output data in all output channels is computed.

15. The apparatus of claim 12, wherein the set of output data corresponds to the maximum size of the output data that can be received at one time in each output channel, and wherein the set of output data includes the complete output data of each output channel or only a portion of the output data of each output channel.

16. The device of claim 15, wherein a portion of the output data of each output channel corresponds to a row or a set of rows for each output channel.

17. The device of claim 15, wherein the processor is configured to: When the set of output data includes only the portion, for the same portion, the preloading of convolutional kernels different from the previously preloaded kernels is repeated, and the control is repeated until all the output data for the portion is computed for all output channels; and For the other parts, restart the preloading process until all the output data in all other parts of all the output channels have been calculated.

18. The apparatus of claim 12, wherein the processor is configured to: allocate a memory location in the computing unit of a buffer memory of a size equal to the minimum size threshold when the size of the available memory location in the temporary working memory is less than a minimum size threshold, and preload a maximum number of convolutional kernels that can be stored at the minimum size threshold in the allocated memory location.

19. The apparatus of claim 12, wherein the convolutional kernel includes weight data, and wherein the processor is configured to: reorganize the weight data of the convolutional kernel during preloading, and optimize the computation of the output data by the computing unit.

20. The apparatus of claim 12, wherein the convolutional kernel includes weight data, and wherein the computing unit is configured to: compute the output data by multiplying and summing the input data of the loaded block with the weight data of the preloaded convolutional kernel.

Images