Data processing method, microcontroller system and computer program product

Abstract

The present disclosure relates to data processing methods, microcontroller systems and computer program products. A computer-implemented method applies a pooling operator to an input array of data, the pooling operator having a set of absorption element values and a set of pooling parameters. A size of an output buffer is computed according to the set of pooling parameters. Elements of the output buffer are initialized to the values of the absorption elements of the pooling operator. For a number of iterations associated with respective pooling windows, an output array of data is generated by: associating, according to the pooling parameters, elements of the input array of the pooling window with output elements of the output buffer; and for each output element of the output buffer, combining respective input elements associated with the output element. Combining can include determining a combination of the respective element of the output buffer and the input elements associated with the output element.

Description

Technical Field

[0001] This specification relates to memory allocation methods, and more specifically to systems and memory allocation methods in artificial neural network (ANN) circuits.

[0002] One or more embodiments relate to a processing circuit device that includes a local memory circuit block and is adapted to implement the ANN processing, such as a microcontroller unit (MCU).

[0003] One or more embodiments can be applied to hardware accelerators, for example, to accelerate processing in artificial neural network circuits. Background Technology

[0004] Artificial neural network (ANN) circuits consist of computational frameworks configured to process large datasets.

[0005] As discussed in this article, the terms "dataset" or "data array" primarily refer to digitized data (e.g., digital images) with an array data structure (or simply array), which comprises a collection of elements (values ​​or variables) identified by at least one array index or key in a manner known per se. Linear arrays (also known as one-dimensional arrays) are a common type of data structure where the input size can refer to the length of the linear array.

[0006] ANN processing generally includes two stages:

[0007] During the training phase, where the ANN adaptively "learns" to perform a task by considering examples (called the training dataset), it typically does not require programming with any task-specific rules; and

[0008] In the inference phase, the trained ANN can be used to make broad predictions on general input data.

[0009] For example, consider classifying an image as either a dog or a cat based on the length and height of the object represented in the image. The training set could include thousands of arrays of [height, weight, cat / dog], such as [fifty, six, cat]. An artificial neural network could utilize this data and "learn" the complex boundaries between cats and dogs based on height and weight. Then, given unclassified data input, a machine learning algorithm determines whether it is a dog or a cat and the confidence level for the classification.

[0010] Various ANN models can be appropriately trained for different applications in different fields. For example, ANNs can be used to learn word patterns or keyword recognition in sentences to prevent tool failures such as in predictive maintenance, for signal processing such as human activity recognition (e.g., inertial sensors), for image processing and object classification in images, and also for many other learning-based tasks.

[0011] Generally speaking, ANNs can be arranged in multiple "layers" and different types of data processing operations can be performed at different ANN layers.

[0012] The execution speed of this data processing operation may depend to a large extent on how the software code is mapped to the computing and memory resources used.

[0013] Applying appropriate operators (or functions) to the data to be processed, the types of data processing operations (or functions) that can be performed at the ANN layer can include applying pooling and / or convolution operators.

[0014] Pooling operators can be applied to data to compute “combination” operations on regions of the provided input data.

[0015] Specifically, pooling is applied to an array to process data within a given "(pooling) window" or "(pooling) region" that is progressively moved / slid across the input area / region based on a certain step forward (e.g., the distance of each slide traversal), where a single output value is computed based on the data collected within the sliding window.

[0016] There are many types of pooling layers or operators, such as the known ones:

[0017] Applying the "max pooling" operator to the input array produces an output array whose elements are obtained by detecting the maximum value within a window at each stage, the window sliding sequentially across the array at each stage; and

[0018] Applying the "average pooling" (avg-pool) operator to the input array produces an array whose elements are obtained by averaging the array values ​​within a window that slides sequentially across the array at each stage.

[0019] You can customize the pooling layer to vary its known parameters, for example:

[0020] Pooling window size / shape, for example (c);

[0021] Fill in the size, for example (p); and

[0022] Stride size, for example (s).

[0023] As mentioned, stride is the length of the "walk distance" (in terms of array index) of the pooling window in the middle of the output value calculation.

[0024] The padding size relates to how array “edge” computations can be handled when pooling is applied: for example, if the input array is a linear array of even length, such as 8, and the pooling window has a size (c) = 3 and a stride (s) = 3, there is a problem at the “end” of the array when moving the window to the last block of data because there are index “missing” values. To avoid this, padding size (p) = 1 increases the height and width of the output, uses null values ​​to fill excess space, and ensures that the pooling window is applied to the complete input.

[0025] Programming a computer to perform pooling operations can involve allocating memory space, also known as buffer size, to host temporary data.

[0026] Improving the memory footprint and computation speed of pooling operations is a crucial quality factor for artificial neural networks, especially for CNN hardware accelerator circuits. Summary of the Invention

[0027] In one embodiment, a method includes: applying a pooling operator to an input array of data, the pooling operator having a set of absorber element values ​​and pooling parameters; applying the pooling operator to the input array of data includes: calculating the size of an output buffer based on the set of pooling parameters; initializing the elements of the output buffer to the values ​​of the absorber elements of the pooling operator; and generating an output array of data stored in the output buffer, wherein generating the output array of data for multiple iterations associated with a corresponding pooling window includes: associating elements of the input array of the pooling window with output elements of the output buffer based on the pooling parameters; and for each output element of the output buffer, combining the corresponding input elements associated with that output element. In one embodiment, the combining includes: determining a combination of corresponding elements of the output buffer with input elements associated with the output elements.

[0028] In one embodiment, a microcontroller system includes: a memory; and processing circuitry coupled to the memory, wherein the processing circuitry applies a pooling operator to an input array of data in operation, the pooling operator having a set of absorber element values ​​and pooling parameters. Applying the pooling operator to the input array of data includes: allocating a portion of the memory to an output buffer, the output buffer having a size that is a function of the set of pooling parameters; initializing elements of the output buffer to the values ​​of the absorber elements of the pooling operator; and generating an output array of data stored in the output buffer. For multiple iterations associated with a corresponding pooling window, generating the output array of data includes: associating elements of the input array of the pooling window with output elements of the output buffer according to the pooling parameters; and for each output element of the output buffer, combining the corresponding input elements associated with that output element. In one embodiment, combining includes: determining a combination of corresponding elements of the output buffer with input elements associated with the output elements.

[0029] In one embodiment, a non-transitory computer-readable storage medium storing content configures a computing system to implement a method comprising: applying a pooling operator to an input array of data, the pooling operator having a set of absorber element values ​​and pooling parameters; applying the pooling operator to the input array of data comprising: calculating the size of an output buffer based on the set of pooling parameters; initializing the elements of the output buffer to the values ​​of the absorber elements of the pooling operator; and generating an output array of data stored in the output buffer, wherein generating the output array of data for multiple iterations associated with a corresponding pooling window comprises: associating elements of the input array with output elements of the output buffer based on the pooling parameters; and for each output element of the output buffer, combining the corresponding input elements associated with that output element. In one embodiment, combining comprises: determining a combination of corresponding elements of the output buffer with input elements associated with the output elements. Attached Figure Description

[0030] One or more embodiments will now be described by way of non-limiting example only, with reference to the accompanying drawings, in which:

[0031] Figure 1 and Figure 2 This is an example diagram of artificial neural network processing;

[0032] Figure 3 It is a diagram of a microcontroller circuit according to one or more embodiments;

[0033] Figure 4A Including parts a), b), and c), a graph of the signals involved in the first pooling method;

[0034] Figure 4BIncluding parts a), b), and c), a graph of the signals involved in the second pooling method;

[0035] Figure 4C Includes parts a), b), and c), including graphs related to buffer loading errors;

[0036] Figure 5 Includes portions a), b), c), d), and e), including diagrams of principles according to one or more embodiments;

[0037] Figure 6 This is a diagram based on the data processing method disclosed herein;

[0038] Figures 7 to 9 Includes example diagrams illustrating the principles according to one or more embodiments;

[0039] Figure 10 This is an example diagram of a two-dimensional input array;

[0040] Figure 11 These are example diagrams illustrating the principles of one or more embodiments;

[0041] Figure 12 These are example diagrams of one or more embodiments of the methods according to this disclosure (e.g., as applied to a two-dimensional image);

[0042] Figure 13 These are example diagrams illustrating one or more embodiments of the methods according to this disclosure;

[0043] Figure 14 It is a schematic diagram based on one or more embodiments; and

[0044] Figure 15 and Figure 16 This is an example diagram illustrating the principles of one or more embodiments. Detailed Implementation

[0045] In the following description, one or more specific details are set forth to provide a thorough understanding of examples of embodiments of this specification. These embodiments may be obtained without one or more of these specific details, or by other methods, components, materials, etc. In other instances, known structures, materials, or operations have not been detailed or described in order to avoid obscuring certain aspects of the embodiments.

[0046] References to "embodiment" or "one embodiment" within the framework of this specification are intended to indicate that a particular configuration, structure, or feature described with respect to that embodiment is included in at least one embodiment. Therefore, phrases such as "in an embodiment" or "in one embodiment" that may appear in one or more locations within this specification do not necessarily refer to the same embodiment.

[0047] Furthermore, in one or more embodiments, specific configurations, structures, or characteristics can be combined in any suitable manner.

[0048] The reference numerals used herein are for convenience only and are not intended to limit the scope of protection or the scope of the embodiments.

[0049] The accompanying drawings are simplified and not drawn to scale. For simplicity, terms of direction (up / down, etc.) or motion (forward / backward, etc.) may be used relative to the drawings.

[0050] The term “coupling” and similar terms do not necessarily mean a direct and immediate connection, but also include connections made through intermediate elements or devices.

[0051] Furthermore, in the following text, the circuit that implements (via HW and / or SW) a neural network (i.e., an artificial neural network (ANN) circuit) may be simply referred to as a "neural network" to avoid making the immediate description too complicated.

[0052] As described above, ANN processing 100 may include multiple data processing operations performed on the input array.

[0053] The data processing operation can be performed automatically using at least one processing circuit (e.g., a microcontroller circuit unit).

[0054] Figure 1 The diagram illustrates a possible graph of an Artificial Neural Network (ANN) 100, including:

[0055] Input layer 102, having a first input node 110 and a second input node 120, is configured to receive an input array Xin, for example, x = (Xin(1)Xin(2)) = (length, height); and

[0056] Output layer 104, having a first output node 130 and a second output node 140, is configured to provide at least one output array Xout to a user circuit, for example, output Xout = (Xout(1), Xout(2)) = (cat, dog).

[0057] ANN processing nodes 110, 120, 130, and 140 may include processing units, such as a "perceptron" in multilayer perceptron processing, and / or a unit or multilayer unit in recurrent neural network processing.

[0058] In the following discussion, for the sake of simplicity, one or more embodiments will be discussed primarily with regard to perceptron-type processing units; otherwise, it should be understood that this type of ANN processing unit is purely exemplary and in no way limiting.

[0059] ANN processing nodes / units 110, 120, 130, and 140 can be configured to process received data using corresponding sets of weights W1, W2, W3, W4 and activities G1, G2, G3, and G4, which can be stored in the memory circuit portion of the processing circuitry used to perform ANN processing.

[0060] In one or more embodiments, the ANN processing 100 may also include so-called “hidden layers” in which the perceptrons are coupled to other neurons in the network and therefore cannot be directly accessed from the input and output layers. This suggests that processing can take advantage of a higher number and more complex architecture of perceptrons compared to processing in a single layer, for example, to process two-dimensional images.

[0061] Cells 110, 120, 130, and 140 of layers 102 and 104 can be coupled to the input node of each cell in the downstream layer (which may be referred to as a “fully connected feedforward” topology) and optionally coupled to the bias input node.

[0062] In one or more embodiments, the ANN processing framework 100 can be modeled as a directed acyclic graph (DAG).

[0063] For the sake of simplicity, regarding Figure 2 The following discussion includes a DAG model with six consecutive nodes; otherwise, it should be understood that such a number of nodes and such a DAG topology are purely exemplary and in no way limiting. One or more embodiments may relate to an ANN processing framework 100, which can be modeled as a DAG with different numbers of nodes and different topologies.

[0064] exist Figure 2 In one or more of the illustrated embodiments, the directed acyclic graph (DAG) may include a set of data processing nodes T1, L1, T2, L2, T3, wherein the nodes in the set of data processing nodes T1, L1, T2, L2, T3 may belong to either of the following two sets of nodes:

[0065] Layer nodes L1 and L2 (in) Figure 1 The first set (represented by circles) includes the data processing operators of the artificial neural network 100; and

[0066] Tensor nodes T1, T2, T3 (in Figure 1 The second set (represented by squares) indicates the memory access or memory phase in the neural network processing framework 100.

[0067] Specifically:

[0068] Any node in the first set of layer nodes L1 and L2 includes at least one data processing operator, which is configured to be applied to data in any (processing and / or multiple) layers (cells) 102 and 104 of the ANN processing 100; and

[0069] Any node in the second group of nodes T1, T2, and T3 includes at least one data storage operation linked to the corresponding data processing stages L1 and L2 of the artificial neural network to load perceptron weights W1, W2, W3, W4 and / or activities G3 and G4, and to store the outputs of the linked neural network layer nodes L1 and L2.

[0070] As mentioned, in the considered (non-limiting) example of perceptron nodes 110, 120, 130, 140, the weights W1, W2, W3, W4 and / or activities G1, G2, G3, G4 of the ANN perceptrons 110, 120, 130, 140 in the corresponding ANN layers 102, 104 can be stored in the memory circuit portion of the processing circuitry.

[0071] Figure 3 This is an example diagram illustrating an exemplary computer processing system 10 suitable for implementing one or more parts of an ANN 100.

[0072] Such a system 10 may include microcontroller processing circuitry, such as general-purpose microcontroller processing circuitry. In one or more embodiments, the microcontroller (MCU) 10 may include a collection of processing core or circuitry 12 and memory circuitry portions 14, 16. The MCU 10 may also include:

[0073] Circuit section or interface 18, configured to couple the MCU to other processing units or actuation devices (e.g., referred to as peripheral devices) in a manner known per se; and networking components between the processing units and the memory, to (e.g., via a data bus (for simplicity) Figure 3 (not shown) Exchanges data between the processing unit and the memory; such networking components may include any of several types of well-known bus structures using any of a variety of bus architectures, such as the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) Local Bus, and Peripheral Component Interconnect (PCI).

[0074] Various components connected to the system bus include, but are not limited to, scalable non-volatile memory (e.g., disk-based data storage), video / graphics adapters, user input interface (I / F) controllers configured to connect to one or more input devices (such as keyboards), peripheral interface controllers configured to connect to one or more external peripheral devices (such as printers), and network interface controllers that can be coupled via a network to one or more devices (such as data storage, remote computers running one or more remote applications), which may include Internet cloud, local area network (LAN), wide area network (WAN), and storage area network (SAN).

[0075] The processing core or circuitry 12 may include one or more general-purpose CPU cores and optionally one or more dedicated cores (e.g., DSP cores, floating-point, GPU, and neural network optimization cores), wherein the one or more general-purpose cores execute general-purpose opcodes, while the dedicated cores perform functions specific to their purpose.

[0076] The set of memory circuit blocks 14 and 16 may include cache registers, dynamic random access memory (DRAM) or extended data output (EDO) memory, or other types of memory such as ROM, static RAM, flash memory, non-volatile static random access memory (NVSRAM), bubble memory, etc.

[0077] System 10 can operate in a networked environment via a connection to one or more remote computers. Remote computers may include personal computers (PCs), servers, routers, network PCs, peer devices, or other public network nodes, and typically include many or all of the elements discussed above.

[0078] Note that other digital computer system configurations may also be used to implement the systems and methods of this disclosure, up to the extent that a particular system configuration is capable of implementing the systems and methods of this disclosure.

[0079] In one or more embodiments, the set of memory circuit portions 14, 16 may include a first memory circuit portion 14 (e.g., RAM-type memory) configured to provide data to a user circuit at a first memory access speed and a second memory circuit portion 16 (e.g., flash memory) configured to provide data to a user circuit at a second memory access speed different from the first memory access speed.

[0080] In one or more embodiments, the first memory portion 14 may include multiple registers or buffers, for example:

[0081] The first buffer subset 140 is configured to store a dedicated data portion, such as the input dataset for an ANN;

[0082] The second buffer subset 142 is configured to store the ANN processing code portion; and

[0083] The third buffer subset 144 is configured to store the activities G1, G2, G3, and G4 of the ANN perceptron nodes 110, 120, 130, and 140 of the corresponding ANN processing layers 102 and 104, where the activities are the outputs of the layers.

[0084] In one or more embodiments, the second memory portion 16 may include an additional plurality of registers or buffers 160, which may be configured to store weight values ​​W1, W2, W3, W4 of the ANN perceptron nodes 110, 120, 130, 140 of the respective ANN processing layers 102, 104.

[0085] In one or more embodiments, activities can be computed on the fly while performing data processing.

[0086] To accelerate the execution of ANN operations and improve the performance of the MCU system 10, buffers can be pre-allocated before the runtime of the ANN 100, for example, before the data to be processed is provided to the ANN input layer nodes.

[0087] Italian patent application No. 102019000012609, filed by the applicant on July 22, 2019, discusses a computer-implemented memory allocation method that can be applied to one or more embodiments to improve the performance of MCU system 10 when applying ANN processing 100 to input array Xin.

[0088] Such computer-implemented methods can include:

[0089] A data processing circuit device is provided for the artificial neural network circuit 100, the data processing circuit device including data buffers T1, T2, and T3 in the memory circuit device;

[0090] The data buffers T1, T2, and T3 in the memory circuit device are coupled to the corresponding processing layers L1 and L2 in the artificial neural network circuit 100, and the ordered access sequence of the layers of the artificial neural network 100 is obtained.

[0091] Based on the ordered access sequence of the layers of the artificial neural network 100, a (virtual) memory allocation schedule is generated. Memory allocation schedule A includes a set of instructions for memory allocation and deallocation operations, applicable to data buffers T1, T2, and T3 in the memory circuit devices 14 and 16 coupled to the processing layers of the artificial neural network circuit 100; and

[0092] The (physical) memory configuration data structure for the artificial neural network circuit 100 is calculated based on the memory allocation schedule. The physical (or virtual) memory configuration dataset includes the size and address of the physical memory locations for the artificial neural network circuit 100.

[0093] In one or more embodiments, as a result of, for example, removing an intermediate buffer, merging two layers of operators can provide further memory savings.

[0094] As mentioned, this paper discusses a (further) improved method for applying artificial neural network processing 100, particularly when using MCU circuit 10, in order to simplify processing and reduce memory access time.

[0095] For simplicity, the principles according to one or more embodiments will be discussed with reference to exemplary cases, wherein:

[0096] The input array Xin is a linear input array with a given length L, such as L = 9; for simplicity, the input array Xin is kept to include unitary vectors, which are vectors whose elements are all unitary values, such as Xin = [A, B, C, D, E, F, G, H, I] = [1, 1, 1, 1, 1, 1, 1, 1, 1];

[0097] The input array Xin is processed using the pooling operator PF, which has a given pooling window PW. The pooling window PW has a pooling window size c and a pooling stride size s. For example, c(PF) = 6 and s(PF) = 6.

[0098] At least one buffer is associated with the pooling operator PF and configured to store the input and / or output of the pooling operator, as discussed below;

[0099] The output signal Xout provides the value of the pooling operator when computation is performed on a portion of the input data equal in size to the pooling window PW. This pooling operator is a "summation" pooling operator; the output of PF is the sum of all elements within the pooling window (e.g., the average pooling operator can be obtained by dividing the output of the summation pooling operator by the input length value L); and

[0100] The operations of reading / writing / summing data can be synchronized via a temporary signal CK (e.g., a physical clock signal or a cycle / period index), and data loaded at one cycle is available at subsequent discrete cycles in a manner known per se, wherein such cycles can be indicated as "epochs".

[0101] In the first "naive" approach of applying pooling to the input array Xin, such as Figure 4AAs shown, pooling can be performed using a first buffer B1(1) with the minimum allocated buffer space (e.g., one memory cell). However, as discussed below, this solution may involve high computational costs in terms of RAM memory size and computational latency.

[0102] The first "naive" implementation of pooling can be conceived as calculating the pooled output value as long as all input values ​​within the pooling window are available for accumulation.

[0103] like Figure 4A As shown, the pooling operator PF implemented using the first "naive" method can include:

[0104] like Figure 4A As shown in part a), a pooling window is applied to the input data Xin in subsequent windows PW1, PW2, PW3, PW4, where each window is separated from the subsequent window by an index element distance equal to the stride size (e.g., the distance of one element).

[0105] The corresponding data windows are "scanned" and each value included in each window is accumulated in memory B1(1), which may store time values ​​such as Figure 4A Part b) is shown; and

[0106] At the end of each accumulation cycle, an output signal Xout is provided, such as Figure 4A In part c).

[0107] like Figure 4A As shown, using the first "naive" method, a number of clock cycles equal to the pooling window size *c* is needed to provide a single output value of the pooling operator *PF*. The total number of output values ​​of the pooling operator *PF* requires a certain number of clock cycles to output. This number of clock cycles can be the product of the pooling window size and a factor that depends on the overlap between subsequent pooling windows *PW1*, *PW2*, *PW3*, and *PW4*. For example, in the considered example, the total computational latency of applying the pooling operator *PF* is 4 * *c* = 24 clock cycles.

[0108] like Figure 4B The second method of applying the pooling operator shown (which can be indicated as a “striped pooling implementation” SF) may include allocating a temporary buffer B2, which has a size equal to the width of the image / array (in the exemplary one-dimensional case, consider it to be equal to the size c of the pooling window PW, e.g., size(B2) = c), and the buffer B2 is configured to store data within pooling windows PW1, PW2, PW3, PW4). Such a striped buffer B2 can be used as a temporary buffer to store rows / columns of the pooling window PW of the input data Xin.

[0109] like Figure 4BAs shown, the striped pooling operator SF implemented using striped pooling can include:

[0110] like Figure 4B As shown in part a), a pooling window is applied to the input data Xin in subsequent windows PW1, PW2, PW3, PW4, where each window is separated from the subsequent window by an index element distance (e.g., a distance of one element) equal to the stride size s.

[0111] The corresponding data window is "scanned," and the values ​​included in the first window PW1 are sequentially loaded into the subsequent index memory portion B2(i) of the stripe temporary buffer B2, such as... Figure 5 Part b) shows a possible timeline of the values ​​stored in the i-th memory portion B2(i) of the striped temporary buffer B2; and

[0112] Whenever all data belonging to the corresponding pooling windows PW1, PW2, PW3, and PW4 is loaded and available for accumulation in subsequent buffer units B1, an output signal Xout is provided to calculate the pooling operator at the end of each accumulation cycle, such as... Figure 4B Part c) is shown.

[0113] In one or more embodiments, at a time that is linearly proportional to the pool size value, such as at time c+3, the output value Xout can be used for further processing.

[0114] In one or more embodiments, stripes are treated as a circular buffer for rows to avoid duplication.

[0115] This striping implementation may be limited because not all pooling operator parameter values ​​can be used appropriately, thus requiring tuning of the pooling operator parameter values. Problems may arise when the pooled image data is at the "edge" of the image, as some stripes of the input data may be left to be stored in the striping buffers B2, B2'.

[0116] like Figure 4B As shown, using this second "striping" method, a number of clock cycles equal to the pooling window size c is required to provide the first output value of the pooling operator PF. The total output value of the pooling operator PF requires a total number of clock cycles to output, which is proportional to the ratio of the input to the stride size. For example, in the considered example, the total computation latency of applying the pooling operator PF is c + 3 = 9 clock cycles, which helps to reduce computational cost.

[0117] Essentially, between time c and time c+1, at least one element in strip buffer B2 is replaced. Specifically, element A is replaced by element G. Similarly, between time c+1 and time c+2, element B is replaced by element H in strip buffer B2. Therefore, element C is replaced by element I in strip buffer B2.

[0118] Therefore, while the naive implementation suffers from the drawback of repeated evaluation of the input, the striped implementation suffers from the drawback of using temporary storage for the input values.

[0119] Other drawbacks of striping implementation may include:

[0120] A certain amount of temporary memory (e.g., RAM) is used to host a temporary buffer B2 that is proportional to the width of the input; and

[0121] In the case of partial pooling regions, i.e., when the pooling window size and the input data size are not multiples of each other (with optional non-zero padding), the state of the stripes depends on the stride, padding, and size of the pooling region, resulting in the need to handle several "special" cases, leading to a "fail-prone" implementation.

[0122] Figure 4C The diagram shows some examples of configurations involving partial pooling regions in a stripe buffer, where exemplary stripe buffer B2' is treated as a circular buffer for rows to avoid duplication.

[0123] As mentioned, issues may arise when applying pooled data processing, such as when processing image data at the edges of an image, leaving behind stripes of data.

[0124] like Figure 4C As shown in part a), in the first exemplary case, the input size and stride size may lack a common dividend / multiple. For example, this could be the following situation:

[0125] The input size L can have a first input value, for example, L = 8;

[0126] The pool size can have a first pool size value, for example, c = 3; and

[0127] The stride size can have a first stride size, for example, s = 3.

[0128] exist Figure 4CIn the first example shown in part a), when the (striped) pooling process SF (with pool size 3 and stride size 3) is applied to the input data Xin (with input size 8), the final pooling region to be computed is partially filled with data because the ratio of the input data size to the pool size is not an integer, for example, 8 mod 3 = 2. In such an exemplary scenario, the stride buffer B2 will only partially store the "valid" data, and problems may occur when calculating the output value Xout as the sum of valid and invalid values ​​when applying the pooling processes SF and PF.

[0129] like Figure 4C As shown in part b), in the second exemplary case, the stride size can be smaller than the pool size; for example, this could be the following:

[0130] The input size L can have a second input value; for example, L = 6.

[0131] Pool sizes can have the same value, for example, c = 3; and

[0132] The stride size can have a second stride size, for example, s = 2.

[0133] exist Figure 4C In the second example shown in part b), when the pooling SF and PF are applied to the first pooling region, the stride buffer B2' may have a partially filled region that includes invalid values ​​that may be located in any part of the stride buffer B2', for example, to the first element B2'(1) of the stride buffer B2', for example, B2'(1) is equal to the previously written value or an obviously erroneous value, such as B2'(1) = NaN.

[0134] like Figure 4C As shown in part c), in the third exemplary case, the stride size can be much smaller than the pool size; for example, this could be the following:

[0135] The input size L can have a second input value, for example, L = 7;

[0136] Pool sizes can have the same pool size value, for example, c = 5; and

[0137] The stride size can have a second stride size, for example, s = 3.

[0138] exist Figure 4C In the second example shown in part b), the final pooling region can be filled starting in the middle and can be wrapped around the buffer, but the middle element B2'(j) of buffer B2' can still include invalid data due to partial filling.

[0139] Striped pooling SF can use specific parts of software code to use specific code to handle each of these "special cases", providing computer program products with expensive maintenance costs.

[0140] In summary, the memory allocation methods discussed above have any of the following drawbacks:

[0141] When pooling regions overlap, repeated calculations occur, increasing latency and reducing efficiency; or

[0142] Using stripes (buffers containing several rows) to cache results presents the problem of managing different edge cases based on the buffer contents.

[0143] One or more embodiments may be based on the observation that pooling operations can be improved when a relationship can be established between input data element Xin and output data element Xout, finding output elements affected by a given input value element, and facilitating single access to stripes of input data Xin and “active” output elements (e.g., output array indices or coordinates) rather than active input elements (e.g., input array indices or coordinates).

[0144] One or more embodiments may include providing an output buffer B3, the output buffer B3 having a size d o The size d o The pooling operator VF can be computed "offline" or "a priori" based on its parameters c and s. Therefore, as mentioned, the pooling operator VF only computes values ​​in the inferred output region.

[0145] In one or more embodiments, the output buffer size d o It can be calculated in a manner selectable from different options, such as by performing output shape inference processing that includes processing pooling parameters.

[0146] For example, such different options could include:

[0147] First output buffer size d of It can be calculated by executing a certain "floor strategy", which can be expressed as:

[0148]

[0149] in

[0150] c: Pooling window size

[0151] s: stride length

[0152] p left = Fill size on the left edge,

[0153] p right = The fill size on the right edge, and

[0154] d i = Input array length.

[0155] Second output buffer size d oc It can be calculated using a certain "ceiling approximation", which is particularly suitable for non-zero padding sizes and can be expressed as:

[0156]

[0157] The size of the third output buffer is d. op It can be calculated without considering padding and with a padding size of zero, and can be expressed as:

[0158]

[0159] Specifically, the inventors have observed that the value shared by multiple (overlapping) pooling regions is combined with the value of each pooling region.

[0160] Any j-th input element Xin(j) contributes to the computation of the value of some i-th output element Xout(i). Such indices of the i-th element Xout(i) that are “affected” by the input falling within the pooling window can be computed before the pooling operation (because their values ​​are the pooling region) and the input values ​​are “reused” or “collapsed” in each computation cycle.

[0161] This solution helps reduce input loading operations, such as accessing at most once, providing a streaming processing model.

[0162] like Figure 5 As shown, the method 50 for applying this "reverse" pooling operator VF, which performs striped pooling, may include:

[0163] like Figure 5 As shown in part a), a pooling window PW is applied to the input data Xin in subsequent windows PW1, PW2, PW3, PW4, where each window is separated from the subsequent window by an index-element distance (e.g., the distance of one element) equal to the stride size s.

[0164] The corresponding data window is "scanned," and the values ​​included in the first window PW1 are simultaneously accumulated in the subsequent index memory portion B3(i) of the stripe temporary buffer B3, such as... Figure 5Parts b) to e) show possible timelines of values ​​stored in output buffer B3 configured to store four elements B3(1), B3(2), B3(3), B3(4); and

[0165] Whenever data belonging to the corresponding pooling windows PW1, PW2, PW3, PW4 has been accumulated in the output buffer B3, for example, when it has been fully accumulated after optional termination, an output signal Xout(i) is provided. The pooling operator is calculated simultaneously with data loading, such as... Figure 5 Parts b) to e) are shown and discussed below.

[0166] Specifically, in Figure 5 In the examples considered in parts b) to e), for example:

[0167] During the first clock cycle, the first value A of the first element of the input data Xin in the first pooling window PW1 can be loaded into the first element B3 of the output buffer B3 (1);

[0168] In the subsequent second clock cycle, the second element B of the input data Xin in the first pooling window PW1 can be accumulated in the first element B3(1), that is, added to the first value A stored in the first element B3(1), and loaded into the second element B3(2) of the output buffer B3.

[0169] In the subsequent third clock cycle, the third element C of the input data Xin in the first pooling window PW1 can be simultaneously accumulated in the first element B3(1), accumulated in the second element B3(2), and loaded into the third element B3(3) of the output buffer B3.

[0170] In the subsequent fourth clock cycle, the fourth element D of the input data Xin in the first pooling window PW1 can be accumulated into the first element B3(1), the second element B3(2) and the third element B3(3) at the same time, and loaded into the fourth element B3(4) of the output buffer B3.

[0171] In the subsequent fifth clock cycle, the fifth element E of the input data Xin within the first pooling window PW1 can be simultaneously accumulated into the first element B3(1), the second element B3(2), the third element B3(3), and the fourth element B3(4) of the output buffer B3; and

[0172] Finally, in the sixth clock cycle, the sixth element F of the input data Xin in the first pooling window PW1 can be simultaneously accumulated to the first element B3(1), the second element B3(2), the third element B3(3)) and the fourth element B3(4) of the output buffer B3. The first element B3(1) is configured to provide the accumulated result as the output value Xout(1) to the user circuit.

[0173] like Figure 5 As shown in part c), in the seventh clock cycle, the second output value Xout(2) can be provided as a result of the data accumulation within the second pooling window that is being completed in the second element of the output buffer B3(2).

[0174] like Figure 5 As shown in part d), in the eighth clock cycle, the third output value Xout(3) can be provided as a result of the data accumulation within the second pooling window that is being completed in the third element of the output buffer B3(3).

[0175] like Figure 5 As shown in part e), in the ninth clock cycle, the fourth output value Xout(4) can be provided as a result of the data accumulation within the second pooling window that is being completed in the fourth element of the output buffer B3(4).

[0176] It should be noted that the examples discussed above are purely exemplary and not restrictive. As mentioned, the values ​​used are for simplicity only; otherwise, it should be understood that any pool window size, stride size, and input / output size can be used in one or more embodiments.

[0177] As mentioned, pooling operators typically share the composition property with addition operators because they are commutative and have an absorbing element: if an element is combined with an absorbing element, that element does not change.

[0178] exist Figure 6 In one or more embodiments shown, the method 50 for performing pooling processing of VF may include:

[0179] Provide input data Xin, which includes at least one array Xin having an input size;

[0180] Provide a pooling operator PF that has the ability to absorb element values ​​o. i The set of pooling parameters c, s, and p, which includes the pooling window size, pooling stride size, and pooling fill size, such as pooling operators selected from known pooling operators, such as max pooling, average pooling, and linear pooling.

[0181] The buffer size d of the 500 output buffer B3 is calculated based on the set of pooling parameters c, s, and p. oc d of d op For example, as a result of performing output shape inference processing, as discussed above;

[0182] Configure and initialize at least one output buffer B3 of 502, the at least one output buffer B3 having a calculated buffer size d. oc d of d op The output buffer B3 has multiple buffer elements B3(1), B3(2), B3(3), B3(4) as a function of the calculated buffer size; and

[0183] Initialize the buffer elements B3(1), B3(2), B3(3), and B3(4) of the 502 output buffer B3 with the absorber element o of the selected pooling operator PF. i The value of .

[0184] For example:

[0185] When the max-pooling operator PF is used in performing reverse pooling VF, the absorbed element can have a corresponding value, for example, o i =-∞, and normalization can be omitted; and

[0186] When the "average" pooling operator PF is used in performing reverse pooling VF, the absorbed element can have a corresponding value, for example, o i =0, and normalization can include dividing the output Xout by the input size that fills the window.

[0187] In one or more embodiments, method 50 may further include:

[0188] The pooling region selection 504 can include determining the output element X based on the pooling parameters c and s. out (j) input element X in (i).

[0189] In one or more embodiments, performing region selection 504 may include calculating range limits, such as including a starting index x. start and terminating index x end Where the index x is used to start and end the output element j. o The calculation, where the start and end elements are determined using the following expression:

[0190]

[0191]

[0192] Where x i It is the index of the i-th element of the input array Xin.

[0193] In one or more embodiments, method 50 may further include:

[0194] For the corresponding pooling window PW, apply the corresponding elements B3(1), B3(2), B3(3), B3(4) of the output buffer B3 and the linear combination 506 of all the inputs that affect it;

[0195] For all elements of the input array Xin, iteratively perform pooling region selection 504, and repeatedly combine the output elements with the input elements 506;

[0196] Optionally, 508 is used, applying the normalization operator g; and

[0197] The output result Xout, stored in the output buffer B3, is provided to the user circuit by 510.

[0198] One or more embodiments of initializing the output facilitate combining (existing) output values ​​with the input. This can prove particularly advantageous when applying pooling with a “large” padding size, such as when the padding size is greater than the window size p > c. In this case, as mentioned, some output values ​​are independent of the input and may not be used.

[0199] In such Figure 9 In the scenario illustrated, method 50 helps avoid wasting computational resources loading values ​​that fall within empty intervals at the fill boundaries. For example, if an empty interval is detected, pooling processing time can be reduced, thus improving efficiency.

[0200] The final optional normalization helps to provide a flexible approach that is tunable for any pooling function that can be used in different applications.

[0201] Figures 7 to 9 Includes an example diagram illustrating the principle underlying the selection of the execution region 504, which determines the input element Xin(i) that affects the output element Xout(j) based on the pooling parameters c and s of the pooling operator PF used in reverse pooling VF.

[0202] The j-th output index x o The calculation may involve sequentially going through all the indexed elements of the input array Xin and selecting the element with an index that is within the range limit of the calculated interval, and performing a (linear) combination of the input value and its current output value.

[0203] This operation can be represented as:

[0204] out(x o ) = f(in(x i ), °out(x o ))

[0205] In Figure 7 the first exemplary scenario shown in part a) of

[0206] x i ∈ [x o ·S, °(x o +1)·s - 1]

[0207] As a result, the selected pooling regions PW1, PW2 can be adjacent to each other. The i-th input element at index x i contributes only to the index element value of the j-th output element index. For example, this index can be expressed as: where x i is the input element index / coordinate, and x o is the output element index / coordinate.

[0208] As Figure 7 shown in part b) of i , in such an exemplary scenario, assuming an input size of d

[0209] In Figure 8 the second exemplary scenario shown in part a) of Figure 7 , the pooling parameters can be different from those discussed in parts a) and b) of i because the stride size can be less than the pooling window size, e.g., s < c. As a result, applying the pooling process PF can include an overlapping region OW. Thus, the i-th input element at index x o contributes to the value of the j-th output element at index x

[0210] In other words, the output "looks back" through the "pool size" elements to find the inputs that contribute to it, but it only advances through the "stride" elements for each output, as Figure 8 shown in part b) of

[0211] exist Figure 9 In the third exemplary scenario shown in part a), the pooling parameters may differ from those regarding... Figure 7 and Figure 8 The parts discussed in a) and b) are excluded because the stride size can be larger than the pooling window size, for example, s > c. As a result, applying pooling to the PF can include the empty region EW. Therefore, index x i The i-th input element at index x o (represented as:) The value of the j-th output element at position j contributes, even though the interval can sometimes be empty, resulting in "disconnected" regions. For example, if Then index x i The i-th input element at index x o The output at that location makes no contribution.

[0212] In a more general case, including non-zero padding size, for example, p ≠ 0, left padding p left You can use index x i The shift of the i-th input element at position i contributes, while right padding at position p... right The 504 can be implicitly interpreted from the output size, as discussed earlier. In such a scenario, the interval can be restricted to the starting index (e.g., x). start =0) and the ending index (e.g., x end = (size-1)).

[0213] As mentioned, although the discussion is primarily about input arrays with a linear array shape, one or more embodiments may be adapted to process other kinds of input data, such as two-dimensional image arrays.

[0214] Figure 10 It is a two-dimensional input array S in Example diagram, the two-dimensional input array S in The elements or pixels can be processed into an element matrix with Cartesian coordinates Xin, Yin, and its elements can be indicated as x. i y i Therefore, reverse pooling (VF) can be applied to elements in both the horizontal and vertical planes, resulting in corresponding output value indices x. o y o .

[0215] Figure 11 This is another example diagram illustrating the underlying principle of one or more embodiments, in which a two-dimensional inverse pooling operator VF' can be applied to process the input array X. in Y inDifferent data in the moving pooling window PW' can affect the corresponding output value in the output Sout.

[0216] exist Figure 11 In one or more embodiments shown, as a reference to image S in The result of applying reverse pooling to VF' can be used to obtain a two-dimensional array Sout.

[0217] Figure 12 This is an example of the operation of method 50' for performing two-dimensional reverse pooling VF, including:

[0218] Provide input array S in The input array S in It includes at least a first (e.g., horizontal) coordinate array Xin with a first input size Lx and a second (e.g., vertical) coordinate array Yin with a second input size Ly;

[0219] Provide a pooling operator PF that has the ability to absorb element values ​​of O. i And a set of pooling parameters c, s, p, which includes the pooling window size, pooling stride size, and pooling fill size, such as a pooling operator selected from known pooling operators (such as max pooling, average pooling, and linear pooling); and

[0220] The buffer size d of the 500' output buffer B3 is calculated based on the set of pooling parameters c, s, and p. oc d of d op The set of buffers, for example, as a result of performing output shape inference processing, as discussed above; specifically, for a two-dimensional image, calculating the buffer size may include calculating the horizontal size of a first buffer and the vertical size of a second buffer, the buffers may be represented as a matrix of column and row elements.

[0221] In one or more embodiments, method 50' may further include:

[0222] Provides 502 with a calculated buffer size d oc d of d op At least one output buffer B3, the output buffer B3 having multiple buffer elements B3(i,j) as a function of the calculated buffer size;

[0223] Initialize the buffer elements B3(i,j) of the 502 output buffer B3 with the absorber element O of the selected pooling operator PF. i The value; and

[0224] Perform pooling region selection 504', which may include determining the input element indices Xin(i) and Yin(i) that affect the output buffer element B3(i,j) based on the pooling parameters c and s.

[0225] In one or more embodiments, performing region selection 504' may include calculating the range limit value, i.e., the starting index x. start y start The set and terminating index x end y end The set is configured to trigger index x o The start and end points for calculating the value of the j-th output element are determined using the following expression:

[0226]

[0227]

[0228] Where X i It is the i-th index / coordinate element of the first input array Xin, X i It is the i-th index / coordinate element of the second input array Yin.

[0229] In one or more embodiments, method 50 may further include:

[0230] For the corresponding pooling window PW, apply the (linear) combination 506′ of the corresponding elements B3(1), B3(2), B3(3), B3(4) of the output buffer B3 with all the inputs that affect it; specifically, such a combination can be expressed as: o(x, y) = f(o(x, y), i(x i y i ));

[0231] For all elements of the input array Xin, perform pooling region selection 504' iteratively, and repeatedly combine the output element of 506' with the input element;

[0232] Optionally, 508 is used, applying the normalization operator g; and

[0233] The result stored in output buffer B3 is provided to the user circuit by 510.

[0234] exist Figure 13 In one or more of the embodiments shown, reverse pooling processing (VF) can be applied in a “streaming” mode, where reverse pooling VF can be applied to data “on the fly” as in a “first-come, first-served” processing logic.

[0235] For example, such as Figure 13 As shown, the data can be packaged into the following structure:

[0236] The first "front" row S L can be added to a first end of the input, e.g., the bottom end, and can be used to store an initialization value of a selected pooling operator PF;

[0237] The second "front" row S U can be added to a second end opposite to the first end, e.g., the top end, and is used to store a final (normalized) value that can be obtained, e.g., after optional normalization; and

[0238] The value of the output array Sout can be sandwiched between the first end and the second end of the data structure.

[0239] Using such a data structure, a streaming array S0 can be obtained, which helps in the continuous computation of the pooling operator.

[0240] For example, once the initialization is performed on the bottom row S L the pooling can be computed on the middle rows, which uses, for example, the size for the output buffer B3 as discussed previously, and the size can be inferred using the pooling interval. In the example considered, the top row S U can be used to perform and store the optional final normalization.

[0241] Thus, one or more embodiments can help in obtaining a burning front behavior where the input Sin is consumed progressively from the top to the bottom.

[0242] In one or more embodiments, further additional / optional operations (e.g., non - linearity, quantization) can be combined with the normalization step to improve efficiency.

[0243] In one or more embodiments, the normalized values can be stored in a second output buffer or strip to handle pipeline operations, e.g., subsequent convolutional layers, which helps in saving memory.

[0244] In one or more embodiments, the range limit values can include initializing the index / coordinate values of the strip S L to a value k, e.g., x = k.

[0245] For example, in one or more embodiments, while x satisfies a given condition, e.g., x < s, it can be incremented and the computed interval limit value x start 、x end (see, for example Figure 10 ). For example, such interval limit values can be advantageously computed as where the values contributing to the output can change value only after x i is incremented s times.

[0246] One or more embodiments can facilitate a fast implementation using increment, comparison, and reset phases.

[0247] For each value in [x start , x end , its memory access location index and the calculated output can be generated, and the calculation is iterated by incrementing the value again until the set condition is met.

[0248] One or more embodiments can relate to corresponding hardware implementations, wherein:

[0249] The operation of checking that x satisfies a given condition (e.g., x < s) can include providing a register configured to store a threshold and performing a (fast) comparison between the value of x and the stored threshold;

[0250] The iterative calculation can include providing a simple register reset; and

[0251] Memory access can be addressed by a DMA component converting coordinates to a memory address.

[0252] Advantageously, one or more embodiments can exhibit a memory consumption comparable to the memory consumption footprint found using naive methods, especially when using pipelining as Figure 13 shown, where the output is updated immediately.

[0253] One or more embodiments can have an efficiency comparable to the striping method, especially when the stride size s is less than the pool size c.

[0254] Table I below summarizes experimental results indicating the favorable performance that can be provided by applying the computer-implemented methods discussed herein. Specifically, the experiments were performed using the following parameters:

[0255] Zero-padding size, p = 0;

[0256] Fixed pooling window size c = 5;

[0257] Variable stride size s = 5 or s = 2; and

[0258] Input image S in having size Lx = 25 and Ly = 25.

[0259] In the experiments considered, pooling was performed after the Conv2D layer using a 3×3 filter, 8 input and 8 output channels, and padding = 1.

[0260] As shown in Table I, using the method as shown herein can reduce complexity relative to naive pooling PF and reduce memory consumption relative to striping pooling SF.

[0261] Table I: Experiment Summary

[0262] stride No merger simple strip Reverse improve s＝5 MACC 360625 360625 360625 360625 0％ memory 5200 200 1200 200 83％ s＝2 MACC 364225 2437825 364225 364225 85％ memory 6352 1352 2352 1352 43％

[0263] One or more embodiments may be applied to and used to improve the performance of convolution operations.

[0264] Applying the convolution operator (CF) can be viewed as applying a weighted form of pooling.

[0265] Specifically, average pooling without partial pooling regions is convolution, where the weights all have the same value that is inversely proportional to the window size, for example, weights w0 = w1 = w2 = 1 / c.

[0266] For example, a transposed convolution with stride = 1 can be equivalent to a convolution, which is achieved by weighting the input using filters and accumulating the weights in the output.

[0267] In one or more embodiments, method 50 is used to apply, such as Figure 14 The convolution operator CF shown can include converting the input value x i Multiply by the corresponding (filter) weight coefficients ω2, ω1, ω0.

[0268] As shown in this article, applying the convolution operator CF can include:

[0269] Use a relative output index x, as can be represented as follows. r Access the weight coefficients ω2, ω1, ω0 in reverse order (for example):

[0270] x r =x o -x start

[0271] in

[0272] x o It is the index of the j-th output element, and

[0273] x start It is the index of the input element being considered.

[0274] If index x o If the j-th output element at position j is outside the range of valid output indices, then the calculated relative output index x can be discarded. r .

[0275] When the stride size s > 1, the relative output index can be represented as:

[0276] x r =S·(x o -x start )+xo In one or more embodiments, mods can use multiple relative output indices, such as a first relative output index x. r Second relative output index y r It can be calculated and used to access the corresponding (two-dimensional) filter coefficients.

[0277] For example, such as Figure 15 and 16 As shown, applying a convolution with a filter W (e.g., W = (ω0, ω1, ω2) = (1, 2, 3)) corresponding to a pooling window size c = 3 with weights ω2, ω1, ω0, the filter W will be convolved with the input vector X. in (For example, X) in =(X in [0], X in [1], X in [2], X in [3], X in [4] = (1, 1, -1, -1, 1)) convolution, the application of convolution involves a stride size s = 1.

[0278] like Figure 15 and Figure 16 As shown, performing a convolution on Xin and W may result in the output vector having three values, for example, Xout = (W·X in [0∶2],W·X in [1∶3],W·X in [2∶4]=(X out (0), X out (1), X out (2))=(0,-4,0).

[0279] For example, such as Figure 15 As shown:

[0280] First input value X in (0) Effects with index X out The output of (0), where the first input value is multiplied by the first weight value ω0;

[0281] Second input value X in (1) Influence X out (0), X out (1), where the second input value X in (1) When multiplying by the second weight value ω1, the first output value X is... out (0) contributes and, when multiplied with the first weight ω0, affects the second output value X. out (1) Made contributions;

[0282] Third input value X in (2) Affecting the output element Xout (2) X out (1) X out (0), contributing to the already calculated first and second outputs, therefore X out (0) now has a final value and can be output by the buffer;

[0283] Fourth input X in (3) Can affect index X out (1) X out (2) X out (3), but the output has a size c = 3, so only X out (1) X out (2) Affected, while X out (3) Discarded; only for X out (1) and X out (2) Each has its own contribution, therefore X out (1) It has a final value; and

[0284] Fifth input X in (4) Only affects X out (2) Give the following for X out (2) The final output value.

[0285] In one or more embodiments, inverting the order in the filter coefficients W can help speed up computation because the lower output index to be accessed is the earliest index, and therefore the computation of the filtered output is completed faster for that index.

[0286] like Figure 16 As shown, the method 50 for applying this "reverse" convolution operator CF, which performs striped pooling, may include:

[0287] like Figure 16 As shown in part a), a convolutional window CW is applied to the input data Xin in subsequent windows CW1, CW2, CW3, where each window is separated from the subsequent windows by an index-element distance equal to the stride size s (e.g., distance s = 1).

[0288] The corresponding data window is "scanned," where the values ​​included in the first window CW1 are simultaneously accumulated in the subsequent indexed memory portion B4(i) of the temporary buffer B4, such as... Figure 16 Parts b) to d) show possible timelines of values ​​stored in output buffer B3 configured to store four elements B4(1), B4(2), B4(3); and

[0289] Whenever data belonging to the corresponding convolutional windows CW1, CW2, CW3, CW4 has been accumulated in the output buffer B4, for example, after optional termination and full accumulation, an output signal Xout(i) is provided. The pooling operator is computed simultaneously with data loading, such as... Figure 16 Parts b) to d) are shown and discussed below.

[0290] Specifically, in Figure 16 In the examples considered in parts b) to d), for example:

[0291] During the first clock cycle, the first value Xin(0) of the first element of the input data Xin in the first window CW1 can be loaded into the first element B4(1) of the output buffer B4 by multiplying by the first weight w0.

[0292] In the subsequent second clock cycle, the second element Xin(1) of the input data Xin in the first window CW1 can be accumulated in the first element B4(1) by weighting with w2, that is, added to the first value Xin(0)w0 stored in the first element B4(1), and loaded into the second element B4(2) of the output buffer B4 by weighting with w1.

[0293] In the subsequent third clock cycle, the third element Xin(2) of the input data Xin in the first window CW1 can be simultaneously accumulated in the first element B4(1) by weighting with w2, accumulated in the second element B4(2) by weighting with w1, and loaded into the third element B4(3) of the output buffer B4 by weighting with w0.

[0294] In the subsequent fourth clock cycle, the fourth element Xin(3) of the input data Xin within the first window CW1 can be simultaneously accumulated into the second B4(2) by weighting with w1 and accumulated into the third element B4(3) of the output buffer B4 by weighting with w0; and

[0295] In the subsequent fifth clock cycle, the fifth element Xin(4) of the input data Xin in the first window CW1 can be accumulated to the third element B4(3) of the output buffer B4 by weighting with w2.

[0296] like Figure 16 As shown in parts a), b), and c), the results of the convolution can be provided using a potentially reduced clock cycle.

[0297] Again, it should be noted that the examples discussed above are purely exemplary and not restrictive. As mentioned, the values ​​used are for simplicity only; otherwise, it should be understood that any pooling window size, stride size, and input / output size can be used in one or more embodiments.

[0298] As shown herein, a computer-implemented method (e.g., 50) includes:

[0299] Provide an input array (e.g., Sin; Xin) that has an array size and consists of a set of ordered indexed elements with corresponding element indices;

[0300] Provide at least one of the following: a pooling operator (e.g., PF) having a set of absorbed element values ​​and pooling parameters (e.g., c, s), the set of pooling parameters including a pooling window size (e.g., c), a pooling stride size (e.g., s), and a pooling fill size (e.g., p). left p right At least one of the following, a pooling operator (e.g., PF, VF; VF′) is selected from the set of pooling operators; and

[0301] The pooling operator is applied to the input array as follows:

[0302] Perform output shape inference processing, including calculating (e.g., 500) the output buffer size of the output buffer (e.g., B3) based on the set of pooling parameters;

[0303] Provide (e.g., 502; 502') at least one output buffer (e.g., B3) having a buffer size equal to the calculated output buffer size, and assign initial values ​​to the buffer elements of the output buffer (e.g., B3(1), B3(2), B3(3), B3(4)) equal to the absorb element of the selected pooling operator (e.g., PF);

[0304] Perform pooling region selection (e.g., 504), which includes determining the input elements of the input array (e.g., Xin) that affect the output elements of the output buffer (e.g., B3, Xout) based on the pooling parameters;

[0305] For the corresponding pooling window (e.g., PW), calculate the combination of the corresponding elements of the output buffer (e.g., B3) with the input elements that affect it (e.g., 506);

[0306] For all elements of the input array, iteratively perform pooling region selection (e.g., 504) and compute linear combinations (e.g., 506); and

[0307] The calculated output (e.g., Xout) stored in the output buffer is provided (e.g., 510) to the user circuit.

[0308] As shown in this document, the set of pooling operators includes at least one of the following:

[0309] Max pooling operators have absorbable elements that tend towards the minimum bottom of the numerical representation interval; and

[0310] Average pooling operator, which has the ability to absorb element o i =0 and a normalization factor equal to the size of the input array.

[0311] As shown in this document, the method includes applying (e.g., 508) a normalization operator (e.g., g) to the result stored in the output buffer.

[0312] As illustrated herein, performing a region selection (e.g., 504) involves calculating a range limit value including a start index and an end index, which are configured to begin and end the calculation of output element values, where the start index x start and the ending index x end Represented as:

[0313]

[0314]

[0315] Where X i It is the i-th index of the input array.

[0316] As illustrated herein, performing output shape inference processing includes calculating (e.g., 500) the output buffer size based on the set of pooling parameters, wherein the calculation is optionally performed in at least one of the following ways:

[0317] First output buffer size d of It is represented as:

[0318]

[0319] Second output buffer size d oc It is represented as:

[0320]

[0321] The size of the third output buffer is d. op It is represented as:

[0322]

[0323] As shown in this article, the pooling operators provided include:

[0324] The input array is processed by an artificial neural network, which includes at least one of pooling or convolutional data processing layers.

[0325] As illustrated herein, a microcontroller system includes:

[0326] Memory circuit blocks, including memory portions configured to be allocated or deallocated to the main data buffer; and

[0327] At least one processing circuit is coupled to the memory circuit block and configured to perform the operation of a computer-implemented method.

[0328] As shown herein, a computer program product includes a software code portion that, when executed in at least one processing circuit, configures such at least one processing circuit to perform the operation of a computer-implemented method (e.g., 50).

[0329] It should also be understood that the various individual implementation options shown in the accompanying drawings are not necessarily intended to be employed in the same combinations shown in the drawings. Therefore, one or more embodiments may employ these (otherwise non-mandatory) options individually and / or in different combinations relative to the combinations shown in the drawings.

[0330] Without prejudice to the fundamental principles and without departing from the scope of protection, details and embodiments may vary, even significantly, with respect to what has been described by way of example only. The scope of protection is defined by the appended claims.

[0331] Various exemplary embodiments are summarized below with reference to the accompanying drawings. In one embodiment, a computer-implemented method (50) includes: providing an input array (Sin; Xin) having an array size and comprising a set of ordered indexed elements having corresponding element indices; providing at least one of the following: a pooling operator (PF) having a set of absorbable element values ​​and pooling parameters (c, s), the pooling operator including a pooling window size (c), a pooling stride size (s), and a pooling fill size (p). left p rightAt least one of the following pooling operators (PF, VF; VF′) is selected from the set of pooling operators; the pooling operators (PF, VF; VF′) are applied to the input array (Sin; Xin) by: performing output shape inference processing, including calculating the output buffer size of the output buffer (B3) according to the set of pooling parameters (c, s); providing at least one output buffer (B3) (502; 502′) having a buffer size equal to the calculated output buffer size, and allocating initial values ​​to the buffer elements (B3(1), B3(2), B3(3), B3(4)) of the output buffer (B3). The initial value is equal to the absorb element of the selected pooling operator (PF); the pooling region selection (504) is performed, including determining which input elements of the input array (Xin) affect which output elements of the output buffer (B3, Xout) based on the pooling parameters (c, s); for the corresponding pooling window (PW), the combination of the corresponding element of the output buffer (B3) with the input elements affecting it is calculated (506); for all elements of the input array (Xin), the pooling region selection (504) is iteratively performed and the linear combination (506) is calculated (506); and the calculated output result (Xout) stored in the output buffer (B3) is provided (510) to the user circuit.

[0332] In an embodiment, the set of pooling operators includes at least one of the following: a max pooling operator having an absorbing element that tends toward the minimum bottom of the numerical representation interval; and an average pooling operator having an absorbing element o. i =0 and a normalization factor equal to the size of the input array. In an embodiment, the method includes applying (508) a normalization operator (g) to the result (Xout) stored in the output buffer (B3). In an embodiment, performing region selection (504) includes calculating range limits including a start index and an end index, the start index and the end index being configured to begin and end the calculation of the output index value, wherein the start index x start and the ending index x end Represented as:

[0333]

[0334]

[0335] Where X i It is the i-th index of the input array (Xin, Sin).

[0336] In an embodiment, performing the output shape inference process includes calculating (500) the output buffer size of the output buffer (B3) based on the set of pooling parameters (c, s), wherein the calculation (500) is selectable in at least one of the following:

[0337] First output buffer size d of It is represented as:

[0338]

[0339] Second output buffer size d oc It is represented as:

[0340]

[0341] Alternatively, the size d of the third output buffer when the padding size is zero. op It is represented as:

[0342]

[0343] In an embodiment, providing a pooling operator includes applying an artificial neural network to the input array, the artificial neural network including at least one of a pooling data processing layer and / or a convolutional data processing layer.

[0344] In one embodiment, a microcontroller system includes: a memory circuit block including a memory portion configured to be allocated or deallocated to a main data buffer; and at least one processing circuit coupled to the memory circuit block and configured to perform operations of a computer-implemented method according to any of the methods disclosed herein. In another embodiment, a computer program product includes software code portions that, when executed in at least one processing circuit, configure such at least one processing circuit to perform operations of a computer-implemented method (50) according to any of the methods disclosed herein.

[0345] In one embodiment, a method includes: applying a pooling operator to an input array of data, the pooling operator having a set of absorber element values ​​and pooling parameters; applying the pooling operator to the input array of data includes: calculating the size of an output buffer based on the set of pooling parameters; initializing the elements of the output buffer to the values ​​of the absorber elements of the pooling operator; and generating an output array of data stored in the output buffer. For multiple iterations associated with a corresponding pooling window, generating the output array of data includes: associating elements of the input array of the pooling window with output elements of the output buffer based on the pooling parameters; and for each output element of the output buffer, combining the corresponding input elements associated with the output element. In one embodiment, combining includes: determining a combination of corresponding elements of the output buffer with input elements associated with the output elements. In another embodiment, combining includes applying weights to the input elements.

[0346] In an embodiment, the input array has an array size and a set of ordered indexed elements with corresponding element indices. In an embodiment, a pooling operator is selected from a set of pooling operators, and the set of pooling operators includes at least one of the following: a max pooling operator with an absorber element tending towards the minimum bottom of the numerical representation interval; and an average pooling operator with an absorber element of zero and a normalization factor equal to the input array size. In an embodiment, the set of pooling parameters includes at least one of a pooling window size, a pooling stride size, and a pooling fill size. In an embodiment, the method includes applying a normalization operator to an output array of data.

[0347] In one embodiment, associating elements of the input array includes calculating range limits including the start and end indices of the pooling window. In one embodiment, the start index x start and terminating index x end Determined based on the following:

[0348]

[0349]

[0350] Where X i Here, is the i-th index of the input array, p is the pooling fill size, c is the pooling window size, and s is the stride size. In an embodiment, calculating the size of the output buffer based on the set of pooling parameters includes:

[0351] The size d of the first output buffer is selected according to the following formula. of :

[0352]

[0353] The size d of the second output buffer should be selected according to the following formula.oc :

[0354] or

[0355] When the fill size is zero, the size d of the third output buffer is selected according to the following formula. op :

[0356]

[0357] Where c is the pooling window size, s is the stride size, and p left It is the fill size on the left edge, p right It is the fill size on the right edge, d i It is the length-size of the input array. In an embodiment, the method includes: applying artificial neural network processing to the input array, wherein pooling operators are applied in a pooling data processing layer or a convolutional data processing layer.

[0358] In one embodiment, a microcontroller system includes: a memory; and processing circuitry coupled to the memory, wherein the processing circuitry applies a pooling operator to an input array of data in operation. The pooling operator has a set of absorber element values ​​and pooling parameters. Applying the pooling operator to the input array of data includes: allocating a portion of the memory to an output buffer, the output buffer having a size that is a function of the set of pooling parameters; initializing elements of the output buffer to the values ​​of the absorber elements of the pooling operator; and generating an output array of data stored in the output buffer. For multiple iterations associated with a corresponding pooling window, generating the output array of data includes: associating elements of the input array of the pooling window with output elements of the output buffer according to the pooling parameters; and for each output element of the output buffer, combining the corresponding input elements associated with the output element. In one embodiment, combining includes: determining a combination of the corresponding elements of the output buffer with input elements associated with the output elements. In another embodiment, combining includes applying weights to the input elements.

[0359] In one embodiment, the microcontroller system includes an interface configured to couple the microcontroller system to other processing units or actuation devices. In another embodiment, the microcontroller system includes a bus system coupling processing circuitry and memory to exchange data between the processing circuitry and memory. In another embodiment, the input array has an array size and a set of ordered indexed elements with corresponding element indices. In another embodiment, a pooling operator is selected from a set of pooling operators, and the set of pooling operators includes at least one of the following: a max-pooling operator with absorber elements tending towards the minimum bottom of the digital representation interval; and an average pooling operator with a zero absorber element and a normalization factor equal to the input array size. In another embodiment, the microcontroller system includes a set of pooling parameters, the set of pooling parameters including at least one of a pooling window size, a pooling stride size, and a pooling fill size. In another embodiment, the microcontroller system includes an output array to which a normalization operator is applied.

[0360] In one embodiment, associating elements of the input array includes calculating range limits including the start and end indices of the pooling window. In one embodiment, the start index x start and terminating index x end Determined based on the following:

[0361]

[0362]

[0363] Where X i is the i-th index of the input array, p is the pooling fill size, c is the pooling window size, and s is the stride size. In the embodiment, calculating the size of the output buffer based on the set of pooling parameters includes:

[0364] The size d of the first output buffer is selected according to the following formula. of :

[0365]

[0366] The size d of the second output buffer should be selected according to the following formula. oc :

[0367] or

[0368] When the fill size is zero, the size d of the third output buffer is selected according to the following formula. op :

[0369]

[0370] Where c is the pooling window size, s is the stride size, and p left It is the fill size on the left edge, pright It is the fill size on the right edge, d i It is the length and size of the input array.

[0371] In an embodiment, a storage content configuration computing system for a non-transitory computer-readable storage medium implements a method comprising: applying a pooling operator to an input array of data, the pooling operator having a set of absorber element values ​​and pooling parameters; applying the pooling operator to the input array of data comprising: calculating the size of an output buffer based on the set of pooling parameters; initializing the elements of the output buffer to the values ​​of the absorber elements of the pooling operator; and generating an output array of data stored in the output buffer, wherein generating the output array of data for multiple iterations associated with a corresponding pooling window comprises: associating elements of the input array with output elements of the output buffer based on the pooling parameters; and for each output element of the output buffer, combining the corresponding input elements associated with the output element. In an embodiment, combining comprises: determining a combination of the corresponding elements of the output buffer with the input elements associated with the output elements. In an embodiment, combining comprises applying weights to the input elements. In an embodiment, the input array comprises an array size and a set of ordered indexed elements having corresponding element indices. In an embodiment, the pooling operator is selected from a set of pooling operators, and the set of pooling operators includes at least one of the following: a max pooling operator having an absorber element that tends towards the minimum bottom of the numerical representation interval; and an average pooling operator having an absorber element and a normalization factor equal to the size of the input array. In an embodiment, the set of pooling parameters includes at least one of a pooling window size, a pooling stride size, and a pooling fill size. In an embodiment, the content includes instructions that, when executed by the computing system, cause the computing system to perform the method.

[0372] The various embodiments described above can be combined to provide other embodiments. If it is necessary to employ concepts from various patents, applications, and publications to provide other embodiments, aspects of the embodiments can be modified.

[0373] Based on the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the following claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be interpreted to include all possible embodiments and the full scope of equivalents to which such claims are entitled. Therefore, the claims are not limited by this disclosure.

Claims

1. A computer-implemented method for memory allocation, comprising: A pooling operator is applied to the input array of data in a neural network. The pooling operator has a set of element values ​​to absorb and pooling parameters. Applying the pooling operator to the input array of data includes: The size of the output buffer is calculated based on the set of pooling parameters, wherein the output buffer is a memory buffer; The elements of the output buffer are initialized to the values ​​of the absorb elements of the pooling operator; and Generating an output array of data stored in the output buffer, for multiple iterations associated with the corresponding pooling window, includes: Based on the pooling parameters, associate the elements of the input array of the pooling window with the output elements of the output buffer; and For each output element of the output buffer, the corresponding input elements associated with the output element are combined; Based on the results of applying the pooling operator, the output of the convolutional neural network is generated; The input array described herein has an array size and is indexed; Associating the elements of the input array includes calculating the range limit value, which includes the start index and end index of the pooling window; The starting index and the terminating index Determined based on the following: ， in is the i-th index of the input array, p is the pooling fill size, c is the pooling window size, and s is the stride size.

2. The method of claim 1, wherein the combination comprises: Determine the combination of the corresponding element of the output buffer with the input element, wherein the input element is associated with the output element.

3. The method of claim 1, wherein combining comprises applying weights to the input elements.

4. The method of claim 1, wherein the pooling operator is selected from a set of pooling operators, and the set of pooling operators includes at least one of the following: Max pooling operators have an absorption element that tends towards the minimum bottom of the numerical representation interval; and The average pooling operator has zero absorbable elements and a normalization factor equal to the size of the input array.

5. The method of claim 1, wherein the set of pooling parameters includes at least one of pooling window size, pooling stride size, and pooling fill size.

6. The method of claim 1, further comprising applying a normalization operator to an output array of the data.

7. The method of claim 1, wherein calculating the size of the output buffer based on the set of pooling parameters comprises: The size of the first output buffer should be selected according to the following formula. : The size of the second output buffer should be selected according to the following formula. : ;or When the fill size is zero, the size of the third output buffer is selected according to the following formula. : Where c is the pooling window size, s is the stride size, and p left It is the fill size on the left edge, p right It is the fill size on the right edge, d i It is the length of the input array.

8. The method according to claim 1, comprising: Artificial neural network processing is applied to the input array, and the pooling operator is applied in a pooling data processing layer or a convolutional data processing layer.

9. A microcontroller for memory allocation, comprising: Memory; as well as A processing circuitry device coupled to the memory, wherein the processing circuitry device, during operation, A pooling operator is applied to the input array of data in a neural network. The pooling operator has a set of element values ​​to absorb and pooling parameters. Applying the pooling operator to the input array of data includes: A portion of the memory is allocated to an output buffer, which is a memory buffer and has a size that is a function of the set of pooling parameters; The elements of the output buffer are initialized to the values ​​of the absorb elements of the pooling operator; and Generating an output array of data stored in the output buffer, for multiple iterations associated with the corresponding pooling window, includes: Based on the pooling parameters, associate the elements of the input array of the pooling window with the output elements of the output buffer; and For each output element of the output buffer, the corresponding input elements associated with the output element are combined; Based on the results of applying the pooling operator, the output of the convolutional neural network is generated; The input array described herein has an array size and is indexed; Associating the elements of the input array includes calculating the range limit value, which includes the start index and end index of the pooling window; The starting index and the terminating index Determined based on the following: ， in is the i-th index of the input array, p is the pooling fill size, c is the pooling window size, and s is the stride size.

10. The microcontroller of claim 9, wherein the combination comprises: Determine the combination of the corresponding element of the output buffer with the input element, wherein the input element is associated with the output element.

11. The microcontroller of claim 9, further comprising an interface configured to couple the microcontroller to other processing units or actuation devices.

12. The microcontroller of claim 9, comprising a bus system coupling the processing circuitry and the memory to exchange data between the processing circuitry and the memory.

13. The microcontroller of claim 9, wherein the pooling operator is selected from a set of pooling operators, and the set of pooling operators includes at least one of the following: Max pooling operators have an absorption element that tends towards the minimum bottom of the numerical representation interval; and The average pooling operator has zero absorbable elements and a normalization factor equal to the size of the input array.

14. The microcontroller of claim 9, wherein the set of pooling parameters includes at least one of pooling window size, pooling stride size, and pooling fill size.

15. The microcontroller of claim 9, wherein the processing circuitry applies a normalization operator to an output array of the data during operation.

16. The microcontroller of claim 9, wherein calculating the size of the output buffer based on the set of pooling parameters comprises: The size of the first output buffer should be selected according to the following formula. : The size of the second output buffer should be selected according to the following formula. : ;or When the fill size is zero, the size of the third output buffer is selected according to the following formula. : Where c is the pooling window size, s is the stride size, and p left It is the fill size on the left edge, p right It is the fill size on the right edge, d i It is the length of the input array.

17. A non-transitory computer-readable storage medium, the storage contents of which configure a computing system to implement a method for memory allocation, the method comprising: A pooling operator is applied to the input array of data in a neural network. The pooling operator has a set of element values ​​to absorb and pooling parameters. Applying the pooling operator to the input array of data includes: The size of the output buffer is calculated based on the set of pooling parameters, wherein the output buffer is a memory buffer; The elements of the output buffer are initialized to the values ​​of the absorb elements of the pooling operator; and Generating an output array of data stored in the output buffer, for multiple iterations associated with the corresponding pooling window, includes: Based on the pooling parameters, associate the elements of the input array of the pooling window with the output elements of the output buffer; and For each output element of the output buffer, the corresponding input elements associated with the output element are combined; Based on the results of applying the pooling operator, the output of the convolutional neural network is generated; The input array described herein has an array size and is indexed; Associating the elements of the input array includes calculating the range limit value, which includes the start index and end index of the pooling window; The starting index and the terminating index Determined based on the following: ， in is the i-th index of the input array, p is the pooling fill size, c is the pooling window size, and s is the stride size.

18. The non-transitory computer-readable storage medium of claim 17, wherein the combination comprises: Determine the combination of the corresponding element of the output buffer with the input element, wherein the input element is associated with the output element.

19. The non-transitory computer-readable storage medium of claim 17, wherein the pooling operator is selected from a set of pooling operators, and the set of pooling operators includes at least one of the following: Max pooling operator; and The average pooling operator has a normalization factor that absorbs elements equal to the size of the input array.

20. The non-transitory computer-readable storage medium of claim 17, wherein the set of pooling parameters includes at least one of pooling window size, pooling stride size, and pooling fill size.

21. The non-transitory computer-readable storage medium of claim 17, wherein the content includes instructions that, when executed by the computing system, cause the computing system to perform the method.

Images