Neural network architecture for a systolic processor network and data processing method using a neural network
The neural network architecture for systolic matrix networks addresses inefficiencies in existing CNN implementations by synchronously mixing neuron data within a matrix network of processing elements, improving performance and reducing resource demands.
Patent Information
- Application Number
- FR2022014620
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-12-29
AI Technical Summary
Existing neural network architectures using systolic matrix networks for deep convolutional neural networks (CNNs) are inefficient due to significant processing time spent organizing and grouping intermediate data, leading to high resource demands and slow execution.
A neural network architecture utilizing a matrix network of processing elements with local memory and data propagation circuits that perform lateral forward or backward mixing of neuron data synchronously, allowing for parallel processing and reduced data transfer between neighboring elements.
This approach reduces latency and memory requirements, enhancing processing efficiency and reducing power consumption by minimizing data reorganization and transfer between processing iterations.
Smart Images

Figure 00000038_0000 
Figure 00000038_0001 
Figure 00000039_0000
Abstract
Description
Title of the invention: Neural network architecture for a systolic network of processors and method of processing data using a neural network Technical field
[0001] The present description relates generally to electronic circuits, and in particular to a neural network architecture for a systolic matrix network of processors and a data processing method using such a neural network architecture. Prior art
[0002] Existing neural network architectures used in computer vision and other applications typically rely on the sequential execution of fully connected (i.e., linear) layers of convolutional neurons. In such a standard fully connected neural network, a single "flat" layer of neuron activation values, represented as a vector, is propagated by each layer to the next layer, after multiplying the vector by a single general weight matrix for the layer. Deep neural networks comprise multiple such layers, each of which performs such an operation until an output value is obtained.
[0003] A systolic matrix processor array is a homogeneous network of data processing units, often referred to as processing elements (PEs), each of which is configured to independently calculate a partial result from data received from upstream neighbors, and pass this result to downstream neighbors.
[0004] It has been proposed to use a systolic matrix array of processors as a hardware accelerator for implementing a deep convolutional neural network (CNN). For example, one solution is described in the publication by Chen, Yu-Hsin, et al. "Eyeriss: An energy-efficient reconfigurable ac-celerator for deep convolutional neural networks", IEEE journal of solid-state circuits 52.1 (2016): 127-138.
[0005] However, a disadvantage of existing solutions for using systolic matrix networks of processors to implement CNNs is that they tend to spend a significant proportion of the total processing time organizing and grouping intermediate data, and sometimes reprogramming the PE elements with a sequence of codes specific to each stage and network layer. This results in a solution that is demanding in terms of processing resources and / or relatively slow to execute.
[0006] There is therefore a need for a new neural network architecture, as well as a new neural network execution strategy for execution on PE systolic matrix networks, which provides improved performance over existing solutions, for example in terms of latency. Summary of the invention
[0007] An object of embodiments of the present invention is to address one or more needs of the prior art.
[0008] According to one aspect, an electronic circuit implementing a neural network is provided, the electronic circuit comprising: a matrix network of processing elements, each processing element comprising a data processing circuit and a local memory, each processing element implementing one or more neurons of the neural network, the local memory of each processing element being configured to store neuron data associated with the one or more neurons implemented by the neural network;and data propagation circuitry configured to perform forward or backward lateral mixing of neuron data stored in the local memory of each processing element by propagating, synchronously performed by each processing element, the neuron data to the local memory of each processing element from the local memory of one or more neighboring processing elements of each processing element, wherein each of the processing elements is configured to process, during a first processing iteration, the neuron data from the one or more neighboring processing elements to produce updated neuron data and to store the updated neuron data in the local memory for use in a subsequent processing iteration. ;
[0009] According to one embodiment, the data processing circuit of each processing element is further configured to calculate an intermediate value from the neuron data from the one or more neighboring processing elements.
[0010] According to one embodiment, the neural network comprises an input layer and at least two hidden layers, the input layer and the at least two hidden layers being implemented by the matrix network of processing elements, wherein the data processing circuitry of each processing element is configured to calculate the intermediate value from the neuron data in relation to each of the hidden layers.
[0011] According to one embodiment, the data propagation circuitry is configured to perform forward or backward lateral mixing of the data from neuron stored by the local memory of each processing element by propagating neuron data to the local memory of each processing element from several of the neighboring processing elements during several iterations, each processing element being configured to receive, synchronously at each iteration, the neuron data stored by the local memory of one of the several neighboring processing elements, at least partially overwriting the neuron data stored in its local memory.
[0012] According to one embodiment, the data propagation circuitry is configured to propagate the neuron data to the local memory of the one or more neighboring processing elements of the array according to a translation sequence following an expanding spiral.
[0013] According to one embodiment, the data propagation circuitry comprises, for each processing element, at least one multiplexer configured to choose from which neighboring processing element and / or to which neighboring processing element the neuron data is propagated.
[0014] According to one embodiment, the matrix array of processing elements is configured to receive one or more images as input data.
[0015] According to one embodiment, the data propagation circuitry is configured to perform forward side mixing of the neuron data, and wherein the neuron data of each processing element is an activation state of the one or more neurons implemented by the processing element.
[0016] According to one embodiment, the data propagation circuitry is configured to perform a backward side mixing of the neuron data, wherein the neuron data of each processing element comprises: an activation state of the one or more neurons implemented by the processing element; and / or a parameter representing an error signal.
[0017] According to one embodiment, the neural network is a convolutional neural network.
[0018] According to another aspect, an imaging device is provided comprising a stack formed of an upper level comprising an image sensor and a lower level comprising the above electronic circuit.
[0019] According to yet another aspect, a method of propagating data in a neural network is provided, this method comprising: storing neuron data associated with one or more neurons implemented by the neural network in a local memory of each processing element of a matrix network of processing elements, each processing element of the matrix comprising data processing circuitry and the local memory; calculating, by the data processing circuitry of each processing element, an intermediate value from the neuron data; and performing, by the set of propagation circuits data, of a forward or backward lateral mixing of the neuron data stored by the local memory of each processing element by propagating, synchronously by each processing element, the neuron data to the local memory of each processing element from the local memory of one or more neighboring processing elements of each processing element, wherein each of the processing elements is configured to process, during a first processing iteration, the neuron data from the one or more neighboring processing elements to produce updated neuron data and to store the updated neuron data in the local memory for use in a subsequent processing iteration.
[0020] According to one embodiment, the method further comprises calculating, by the data processing circuit of each processing element, an intermediate value from the neuron data from the one or more neighboring processing elements.
[0021] According to one embodiment, the neural network comprises an input layer and at least two hidden layers, the input layer and the at least two hidden layers being implemented by the matrix network of processing elements, the method further comprising the calculation, by the data processing circuit of each processing element, of the intermediate value from the neuron data in relation to each of the hidden layers.
[0022] According to one embodiment, the forward or backward lateral mixing of the neuron data stored by the local memory of each processing element is performed by propagating the neuron data to the local memory of each processing element from several of the neighboring processing elements during several iterations, each processing element being configured to receive, synchronously at each iteration, the neuron data stored by the local memory of one of the several neighboring processing elements, at least partially overwriting the neuron data stored in its local memory. Brief description of the drawings
[0023] These characteristics and advantages, as well as others, will be explained in detail in the following description of particular embodiments given without limitation in relation to the attached figures among which:
[0024] [Fig.l] illustrates a fully connected neural network according to an exemplary embodiment;
[0025] [Fig.2] schematically illustrates a matrix network of processing elements according to an exemplary embodiment of the present description;
[0026] [Fig.3] schematically illustrates a weighting matrix according to an example of realization of this description;
[0027] [Fig.4] schematically illustrates forward lateral mixing using a matrix array of processing elements according to an exemplary embodiment of the present disclosure;
[0028] [Fig.5] schematically illustrates a 3 by 3 matrix network of processing elements according to an exemplary embodiment of the present description;
[0029] [Fig.6] schematically illustrates a processing element module comprising a processing element and input / output multiplexers according to an exemplary embodiment;
[0030] [Fig.7] schematically illustrates a processing element module comprising a processing element and input / output multiplexers according to another exemplary embodiment;
[0031] [Fig.8] schematically illustrates a forward lateral mixing operation according to an exemplary embodiment of the present description;
[0032] [Fig.9] schematically illustrates memory portions of the matrix array of processing elements of [Fig.8] according to an exemplary embodiment of the present description;
[0033] [Fig. 10] schematically illustrates a translation sequence between local memories of the matrix array of processing elements of [Fig. 8] during forward lateral mixing according to an exemplary embodiment of the present disclosure;
[0034] [Fig.l 1] schematically illustrates a radius of influence of a treatment element for one, two and three layers of lateral mixing according to an exemplary embodiment of the present description;
[0035] [Fig. 12] schematically illustrates an imaging device comprising an image sensor stacked on a data processing chip according to an exemplary embodiment of the present description;
[0036] [Fig. 13] is a flowchart illustrating operations of an inference method involving forward lateral mixing according to an exemplary embodiment of the present disclosure;
[0037] [Fig. 14] schematically illustrates a rearward lateral mixing using a matrix array of processing elements according to an exemplary embodiment of the present disclosure;
[0038] [Fig. 15] schematically illustrates a back-propagation operation according to an exemplary embodiment of the present description;
[0039] [Fig. 16] is a flowchart illustrating operations of a learning method involving backward lateral mixing according to an exemplary embodiment of the present disclosure;
[0040] [Fig. 17] schematically illustrates back-propagation operations of several layers according to an exemplary embodiment of the present description; and
[0041] [Fig. 18] is a graph representing test results according to an exemplary embodiment of the present description. Description of embodiments
[0042] The same elements have been designated by the same references in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same references and may have identical structural, dimensional and material properties.
[0043] For the sake of clarity, only the operations and elements useful for understanding the embodiments described herein have been illustrated and described in detail.
[0044] Unless otherwise indicated, the term "connected" is used to denote a direct electrical connection between circuit elements, while the term "linked" or "coupled" is used to denote an electrical connection between circuit elements which may be direct or via one or more elements.
[0045] In the following description, unless otherwise indicated, when referring to absolute position qualifiers such as the terms "front", "back", "top", "bottom", "left", "right", etc. or relative position qualifiers such as the terms "above", "below", "upper", "lower", etc. or orientation qualifiers such as "horizontal", "vertical", etc., reference is made to the orientation shown in the figures.
[0046] Unless otherwise specified, the expressions "about", "approximately", "substantially" and "of the order of" mean to within 10%, preferably to within 5%.
[0047] [Fig.l] schematically illustrates a fully connected neural network 100 according to an exemplary embodiment.
[0048] The neural network 100 comprises three layers, an input layer (INPUT LAYER), a single hidden layer (HIDDEN LAYER) and an output layer (OUTPUT LAYER). In alternative embodiments, there could be more than one hidden layer. Each layer comprises, for example, a number of neurons. For example, the neural network 100 corresponds to a neural network model with a two-dimensional data space, and there are therefore two visible neurons in the input layer receiving the corresponding input values II and 12. The model has a hidden layer with seven hidden neurons and thus corresponds to a matrix of dimensions R2'"7 In the example of [Fig.l], there are three neurons in the output layer, providing an output O of the network 100.
[0049] In the general case where the neural network comprises £ hidden layers, the function O = H f / l) applied by the neural network 100 is for example an aggregation of functions, comprising an associative function in each layer i, these functions being connected in a chain according to the map O = H^( / ) = HL( ... ( H-, ( ) ... ) )• H y has only two such functions in the simple example of [Fig.l], corresponding to those of the hidden layer and the output layer.
[0050] An output of each neuron of each layer l, except the output layer, is coupled to each neuron of the next layer via a corresponding synapse, the synapses of each layer l being associated with a synaptic weight matrix. In particular, each neuron j of each layer / - 1 of the network 100 is coupled to each neuron k of the next layer / via a corresponding synapse, associated with a synaptic weight VVyjf
[0051] Implementing a neural network such as that of [Fig.l] on a processing device generally involves sequential execution of the linear neural network, but this implies that a significant amount of processing time and resources are occupied in organizing and grouping the intermediate data, with numerous loading and storing operations in memory. This results in a solution that is slow and / or very energy-intensive.
[0052] Another type of neural network is a convolutional neural network, in which each layer involves the application of a 2-dimensional (2D) convolution. Thus, the input states provided to each neuron of each PE element from the neurons of the previous layer no longer form a vector of scalars, but a feature map resulting from the 2D convolution from the corresponding kernel.
[0053] [Fig.2] schematically illustrates a network 200 of processing elements according to a exemplary embodiment of the present description. The network 200 is configured to implement a neural network comprising one or more hidden layers, such as the neural network of [Fig.l]. The network 200 is for example configured to implement a deep lateral mixer neural network, which is a network in which, rather than being a fully connected network, the interconnections of neurons from one layer to the next are defined by the spatial relationship between the neurons, as will be described in more detail below. Such an architecture is for example particularly suited to processing two-dimensional input data such as images, although other applications are also possible.
[0054] In the example of [Fig.2], the matrix 200 is a 2-dimensional matrix of I by J processing elements PE, where I is equal to 8 and J is equal to 6, although in other embodiments, different values of I and J are possible, at least four processing elements being present in the array for example.
[0055] Each of the processing elements implements k neurons, k being for example equal to one or more, and for example equal to at least four. Embodiments described herein are based on the presence of four neurons implemented by each processing element. Each processing element PE, for i E [0, / ] and i [0, J], stores for example a k-element vector hi_j, indicating a hidden state, which will also be referred to herein as an activation state, of the k neurons of the processing element PE. Each processing element PE also stores for example a weight matrix Wq, which defines how the activation states of neighboring processing elements are propagated to a given PE element. In some embodiments, as described in more detail below in connection with Figure 3, each weight matrix Wj j is defined by .
[0056] In the example of Figure 2, each PE element receives, for example, the activation states for neighboring PE elements that share an edge or corner with the PE element in question. Thus, each PE element receives the activation states of up to eight neighboring PEs, in addition to storing its own activation state. An example is shown for processing element PE^ which receives the activation states of processing elements at positions x,y relative to the position of processing element PE5 \ where x and y are each integers from -1 to +1. Thus, the weighting matrix 3 associated with processing element PE53 defines, for example, weights applied to the activation states of each of the hatched PE elements, including element PE53 itself.
[0057] In the case of PE elements that are located at an edge or corner of the array, the weighting matrix is for example reduced in size to cover only the PE elements that are present, or is filled with zeros for PE elements that are not present in the array. It would also be possible for the PE elements to be programmed with an "ignore" flag so that they ignore the processing steps to be performed on the data from the PEs that are not present. As an additional possibility, a ring of dummy PEs is provided around the array, these dummy PE elements provide activation states set to zero and propagate activation states received from neighboring PE elements.
[0058] Of course, the example of dimensions of the weighting matrix Wj j described above and illustrated in FIG. 2 is only an example, and in alternative embodiments, other combinations would be possible. For example, the weighting matrix Wÿ of each PE element could define that, for a given PE element, the activation states are only received by neighboring PE elements sharing an edge with this PE element. As another example, the weighting matrix of each PE element could define that, for a given PE element, the activation states are received from all neighboring PE elements sharing an edge with this PE element and from all PE elements sharing an edge with a PE element that shares itself an edge or corner with the given PE element.
[0059] Each PE element of Figure 2 is for example configured to implement each layer of the DLM neural network by updating the activation state j to a new value hu^, which is then processed by the next layer L. For example, each PE element is configured to apply the following operation: [Math 1] / ** \ = HJ where f() is the activation function.
[0060] Figure 3 schematically illustrates the weighting matrix ^5 3 associated with and stored by the processing element [Math.l] PE53 of [Fig.2] according to an exemplary embodiment in which each PE element implements four neurons, and thus k = 4.
[0061] The weighting matrix Vk5 3 comprises for example nine weighting sub-matrices, W5 3 vv, for x E [-1,0, 1 ] and yg [-1,0, 1 ]. In other words, the weighting matrix is defined by the nine sub-matrices being in the form of 3 by 3 matrix.
[0062] Each of the processing elements of Figure 2 is for example associated with and stores a weighting matrix similar to the matrix 1¾ although in the case of PE on edges or corners of the matrix array, the sub-matrices which are not present in the matrix array are for example zero. In some embodiments, each of the weighting matrices W} j is different from the other weighting matrices. Alternatively, some or all of the weighting matrices Wÿ could be the same, in other words all of the PEs store the same set of weights, with the possible exception of edge / corner cases. Alternatively, the weighting matrices are the same for given regions of the matrix array, such as quadrants or concentric rings centered around the origin, such as the center of the matrix array of PE elements, where each region corresponds for example to a given region of the image plane.
[0063] Each weighting sub-matrix is for example of dimensions k by k*, where k* denotes the number of neurons in each PE element, i.e., the size of each neuron activation vector h;j, in the next layer of the neural network. Thus, each weighting sub-matrix allows the generation of k* activation states from the k activation states coming from the corresponding PE element. In the example of [Fig.3], k* = k, meaning that there is the same number of neurons in the layer associated with the weighting matrix as in the following layer.
[0064] For example, there is a weighting matrix defined for each layer of the neural network that is implemented by the matrix network of PE elements. Alternatively, at least some of the layers may have the same number of neurons and have the same weighting matrix, thereby reducing the number of weighting matrices to be stored by each PE element.
[0065] While the example of Figure 3 relies on a weighting matrix comprising a corresponding scalar weight value applied by each PE element between each of the k input neurons and each of the k* output neurons, in alternative embodiments, the matrix network of PE elements 200 of Figure 2 could be adapted to implement a 2D convolutional network. In such a case, convolutional lateral mixing may be applied, in which the activation state hj^j of each PE element is no longer a 1-dimensional vector, but rather a 2D feature map, and each weighting matrix Wq is no longer a 3 by 3 matrix of k by k* dimensional weighting sub-matrices, but a 3 by 3 matrix of convolution kernels. Furthermore, each PE element is for example configured to apply the following operation: [Math 2] where f() is an activation function and the operator (*) is a convolution operator. The amount of margin and the stride applied during the convolution operation are for example chosen according to whether the size of the feature map is to be increased, reduced, or preserved, from one layer to the next. As known to those skilled in the art, the margin corresponds to a thickness of pixels, whose states are set to zero, which are added around the boundary of each feature map to which the convolution is applied. The stride corresponds to the step size applied between the input feature map and the kernel. A margin of 1 and a stride of 1 result for example in the preservation of the size of the feature map. A stride of 2 and a margin of 1 will decrease the size of the feature map by 4 times.
[0067] [Fig.4] schematically illustrates forward lateral mixing in a deep lateral mixer (DLM) neural network implemented using a matrix network of PE processing elements according to an exemplary embodiment of the present disclosure.
[0068] The matrix array of processing elements has dimensions of four by four in the example of [Fig.4] and each processing element implements four neurons.
[0069] An input layer 400 of the DLM neural network comprises input data in the form of an image in the example of [Fig. 4], although other types of input data are possible in alternative embodiments. Furthermore, in the example of [Fig. 4], the input layer 400 is a square image of 12 by 12 pixels, although in alternative embodiments input images having different dimensions are possible. The input data is for example an image divided into macropixels (MACROPIXELS), the number of macropixels being equal to the number of PEs in the matrix array of PE elements such that each PE element processes the pixel data and the intermediate activation vectors of a macropixel.
[0070] The DLM neural network according to the example of [Fig. 4] comprises three hidden layers 401, 402, and 403 in addition to the input layer 100. In alternative embodiments, it would be possible to have any number of hidden layers. While in [Fig. 4] each of the layers 401, 402, and 403 is represented by a separate matrix network of processing elements PE, this is done only for illustrative purposes to represent the lateral mixing steps, since in reality these layers are all implemented by means of a single matrix network of processing elements PE.
[0071] A first layer 401 of the DLM neural network is for example configured to reduce the size of the input vector associated with each PE element which is the number of neurons implemented by each PE element in the first hidden layer 401. The extent of the reduction is controlled by the shape of the weight matrix. Thus, each PE element applies a weight matrix to a rectangular group of input pixels, where each weight matrix has dimensions kjnpia X k^, where kinput is the number of input pixels processed by each PE processing element, corresponding for example to a macropixel of the image. In the example of Figure 4, kjnpu(= 9, and k = 4, and each weight matrix thus has dimensions of 9x4. Each of the weight matrices of the first layer 401 in the example of [Fig.4] thus reduces the 9 input pixels per PE into a hidden vector of 4. The weights of each weight matrix can for example be learned.
[0072] Alternatively, rather than reducing the size of the input in this way, one can for example use a convolution operation to reduce the size, for example by choosing a step greater than 1. This has the advantage of involving fewer weights. For example, for a 3x3 input and a desired size of kl = 4, a kernel A 2x2 convolution is passed without margin and a stride of 1 to obtain a 2x2 feature map, i.e. a vector of size 4. If the input was much larger, a convolution with a stride greater than 1 and / or a series of convolutions could be applied. The weights of the kernel used for the convolution can for example be learned. Furthermore, the convolution operation could be a convolutional side-shuffling operation.
[0073] It will be noted that, in the example of [Fig. 4], lateral mixing is not performed between the input layer 400 and the first layer 401. Thus, each weighting matrix stored by the PE elements in association with the layer 401 does not include, for example, sub-matrices associated with any neighboring PE element. This is because such a weighting matrix that performs both size reduction and lateral mixing would have relatively large dimensions. However, it would also be possible to perform lateral mixing in the first layer as described in relation to [Fig. 3] if the memory of each PE element is sufficient in size, or if convolutional lateral mixing is performed, as mentioned above.
[0074] [Fig. 5] schematically illustrates a 3 by 3 matrix 500 of processing elements in more detail according to an exemplary embodiment of the present disclosure. The matrix array 500 of only nine processing elements is shown for ease of illustration, and in an alternative embodiment, the matrix array is larger, for example having dimensions of I by J processing elements, where I and J may each be several hundred or more. Furthermore, the array need not be square, but could have rectangular dimensions with I not equal to J.
[0075] Each PE element comprises for example a memory (Mem) 502 storing data. In some embodiments, the memory 502 is a volatile memory, such as an SRAM (static random-access memory) or a register file, for example formed of flip-flops. The memory 502 is for example divided into sections of different sizes which contain data variables, as will be described in more detail below. The data can be written into the memory 502 directly by a sensor if any (not illustrated in [Fig. 5]) and / or by an on-chip memory which is located outside the matrix array of PE elements, and / or by the neighboring PE elements via the interface circuits 503, for example one or more multiplexers, positioned between the PE elements.
[0076] Each PE element also comprises, for example, a processor (ALU) 508, such as an arithmetic logic unit (ALU), configured to address data in the memory 502, to perform calculations on the data such as, for example, multiplication, addition and bit shifting operations and to write the results back to the memory 502 at the desired locations. The way in which the data in the memory 502 is managed and the sequence of operations performed by the processor 508 on this data is defined, for example, by computer instructions forming a computer code, also called microcode, stored in a code memory (Code) 504. In some embodiments, the code memory 504 is implemented by a memory circuit separate from that implementing the memory 502, although it is also possible for the memories 502 and 504 to be implemented by the same memory circuit. Like the memory 502, the memory 504 is for example implemented by a volatile memory. To keep the spatial footprint of each PE element relatively low, the code memory 504 is for example relatively constrained in size and thus limited in the number of instructions it can store.Therefore, in some embodiments, multiple codes may be loaded and executed sequentially to achieve the desired operation of the neural network. For example, the matrix array of PE elements 500 is positioned adjacent to a larger memory circuit and general controller (not shown), which allows data to be written to and read from the local memory circuits 502 and the local code memories 504 of each PE element. It would also be possible, in some embodiments, for the code memory 504 to be implemented by non-volatile memory, such as Flash or EEPROM (Electrically Erasable Programmable Read-Only Memory).
[0077] In some embodiments, the code executed by each PE element is the same, such that each PE element in the matrix array of PE elements performs the same operations synchronously with the others.
[0078] While [Fig.5] relies on a software implementation of each PE element, a hardware implementation would also be possible. In such a case, the code memory 504 and the processor 508 are for example replaced by a single dedicated circuit block configured to repeat predefined operations on the data stored by the memory 502, such as multiplication and accumulation operations which can be parameterized.
[0079] The interface circuit 503 of the matrix network of PE elements 500 allows, for example, the sharing of data from the memory 502 of each PE element with neighboring PE elements. Examples of the implementation of these interfaces will now be described with reference to FIGS. 6 and 7.
[0080] [Fig.6] schematically illustrates a PE module 600 comprising a processing element and / or input / output multiplexers MUX_IN, MUX_OUT, according to an exemplary embodiment. This module 600 is for example used to implement each PE element and the corresponding multiplexers of [Fig.5]. Using the example of module 600 of [Fig. 6], the input multiplexer MUX_IN has eight inputs, one of which is coupled to the memory 502 of each of the eight neighboring PE elements, for example via the output multiplexer MUX_OUT of each neighboring PE module. The output of the input multiplexer MUX_IN is coupled to the PE element, and in particular to the memory 502 of the PE element. The multiplexer MUX_IN is controlled by a selection signal IN_SEL, produced for example by the processor 508 of the PE element. The output multiplexer MUX_OUT of the module 600 has for example one input coupled to the PE element, and in particular to the memory 502 of the PE element, and eight outputs, one of which is coupled to the memory 502 of each of the eight neighboring PE elements, for example via the input multiplexer MUX_IN of each neighboring PE module. The multiplexer MUX_OUT is controlled by a selection signal OUT_SEL, produced for example by the processor 508 of the PE element.
[0081] [Fig.7] schematically illustrates a PE 700 module comprising an element of processing and / or input / output multiplexers MUX_IN, MUX_OUT, according to another exemplary embodiment. This module 700 is for example used to implement each PE element and the corresponding multiplexers of [Fig.5]. In the example of the module 700 of [Fig.7], the input multiplexer MUX_IN has four inputs, one being coupled to the memory 502 of each neighboring PE element which is horizontally or vertically adjacent in the network, for example via the output multiplexer MUX_OUT of each corresponding neighboring PE module. The output of the input multiplexer MUX_IN is coupled to the PE element, and in particular to the memory 502 of the PE element. The multiplexer MUX_IN is controlled by a selection signal IN_SEL, produced for example by the processor 508 of the PE element.The output multiplexer MUX_OUT of the module 600 has, for example, an input coupled to the PE element, and in particular to the memory 502 of the PE element, and four outputs, one being coupled to the memory 502 of each of the eight neighboring PEs, for example via the input multiplexer MUX_IN of each neighboring PE module. The multiplexer MUX_OUT is controlled by a selection signal OUT_SEL, produced for example by the processor 508 of the PE element.
[0082] During the lateral mixing operation, each PE element performs, in parallel, an operation on its activation states and the activation state on each of its neighboring PEs. The PE module 600 of [Fig.6] allows the activation state to be provided to the PE element from each of the eight neighboring PEs and from the PE element to any one of the eight neighboring PEs. The PE module 700 of [Fig.7] allows the activation state to be provided to the PE element from any one of the four horizontally and vertically adjacent PEs and from the PE element to any one of the four horizontally and vertically adjacent PEs. This further allows a given PE element to obtain the activation state of its eight neighbors using a specific translation sequence, as will be described in more detail below with reference to [Fig. 10].
[0083] Although not illustrated in Figures 6 and 7, in some embodiments, an additional circuit interface is provided, for example, on each side of the multiplexers, configured to select the section of the PE memory 502 into which data is written from a neighboring PE element, and / or into which it is read for transmission to a neighboring PE element. For example, in some cases, data may be read and written from one memory to another in blocks that are sent over several cycles, and this still additional interface circuit selects the memory addresses into which these blocks are read and written.
[0084] [Fig.8] schematically illustrates a forward lateral mixing operation according to an exemplary embodiment of the present description.
[0085] The operation represented by [Fig. 8] is for example performed by each PE element of a matrix network of PE elements, such as the matrix networks of Figures 2 or 5. [Fig. 8] illustrates a central PE element 5 and its eight neighbors PE 1 to PE 4 and PE 6 to PE 9, and represents the operations implemented by the central PE element. A similar operation is for example implemented by each PE element of the network.
[0086] It is assumed that, in the first step, the pixel data gathered at a given time, such as a still image, or a frame of a video sequence, are written into a section of the memory of each PE element. For example, as described above in relation to [Fig.4], in the case where the input data is image data, the input image is for example divided into macropixels. In the case where each macropixel comprises 16x16 pixels, then the input data vector for the DLM neural network at each PE element is for example a flattened vector of 256 elements, in which each entry is a number corresponding to a detected pixel intensity. To produce the first hidden state of the DLM neural network for each PE element, this 256 long vector is for example multiplied by a 256xH weighting matrix, for example by applying an inner product or a vector-matrix multiplication.In this example, H = 16 is assumed, although different values are possible. After this operation, each PE element will have, in its local memory 502, a vector having a length of 16 values, hereinafter called the hidden state vector or activation state vector. Assuming that all values are quantized as 4-bit integers, except for the intermediate partial sums, this corresponds to a memory usage of 2.2 kB per PE for the execution of this first step.
[0087] One or more lateral mixing operations, corresponding to one or more hidden layers of the network, are then, for example, performed. During these operations, PE elements, iteratively over a number of lateral mixing layers, update their local hidden state via MAC (multiply and accumulate) operations with the activation states stored in their neighboring PE elements. This exploits the ability of PE elements to translate their memory contents, via multiplexers, between neighboring PE elements, as described above with reference to Figures 6 and 7.
[0088] The lateral mixing operations involve, for example, the processing, by each PE element, of three types of data stored in the local memory 502 of the PE element, as will now be described in more detail with reference to [Fig.9].
[0089] [Fig.9] schematically illustrates memory portions of the matrix network of processing elements PE 1 to PE 9 of [Fig.8] according to an exemplary embodiment of the present description.
[0090] Each PE element is illustrated as containing three memory portions, which correspond for example to three different sections of the local memory 502 in each PE element.
[0091] A first input section (Input) 902 of each PE element contains for example a hidden state vector of the PE element. In some embodiments, the input section 902 initially stores the hidden state vector for the PE element itself, and this value is then overwritten by the hidden state vectors imported by translation from the neighboring PE elements. Alternatively, the input section 902 may comprise two sections, a first which stores the local hidden state of the PE element and a second which stores the hidden state imported by translation from the neighboring PE elements.
[0092] A second weight section (Weights) 904 of each PE element contains for example the weights which will be multiplied by the activation state vector of the PE element and the activation state vector of each of the neighboring PE elements. These weights have for example dimensions of kxk*x9 and each weight is for example a scalar or a convolution kernel. In some cases, for each DLM layer, a new weighting matrix is loaded into each PE element from an external memory. In other embodiments, the same weighting matrix is for example used for some or all of the hidden layers.
[0093] A third output section (Output) 906 stores the cumulative weighted sum as the algorithm iterates through the MAC operations from the activation state vector of the PE element itself and each of the eight neighboring PE elements and their corresponding weight matrices. After all the weighted sums have been accumulated, a non-linear activation function can be applied to this value and the result written to the Input section, to serve as the activation vector that is used for the next DLM layer.
[0094] Referring again to [Fig.8], all PE elements are for example configured to shift their activation state vector, corresponding for example to the input section 902 of their local memory 502, at the same time in the same direction, so that this transfer occurs in parallel across the entire matrix array of PE elements. The translation operation is for example coordinated by a global clock for the entire matrix array of PE elements, so that the PE elements operate synchronously. Assuming a hidden layer size of 16 values, this uses for example a 9x16x16 weight matrix per layer. Assuming 4-bit quantization, this corresponds to a memory usage of 1.2 kB in the local PE memory per layer, or in the case where the same weight matrix is shared between all layers, a memory usage of 1.2 kB for all layers.
[0095] The generation of a set of partial products is represented in [Fig.8] by links between the four neurons of a central PE element 5 and a first neuron of the PE element 5, and by links between the four neurons of each neighboring element PE 1 to PE 4 and PE 6 to PE 9 with the first neuron of the PE element 5. At each neuron, the set of multiplications is accumulated, in other words the result of the current multiplication is added to the result of all the previous multiplications, and stored in an "output" section of the local PE element memory 502.
[0096] As shown in a block 804 in [Fig.8], after obtaining the final weighted sum, the PE element may locally apply a non-linear activation function f(h), such as for example a ReLU, tanh or sigmoid function, to obtain the new activation state vector which is written into the input section 902 of the PE element local memory 502, for example by overwriting the value currently stored therein, which is the activation state vector of the final neighboring PE element which had been imported by translation. Between layers, this vector may also be subjected to a normalization function such as batch normalization, instance normalization or layer normalization, as is commonly practiced in neural networks.
[0097] As represented by the repeat symbol 806 in [Fig. 8], the same sequence of operations is, for example, repeated for a desired number of lateral mixing layers, without any preprocessing or reorganization of data between layers, and in some cases, without loading new instruction codes into the code memory 504 of each PE element. All activation data is in the correct location for use in the next layer, which is typically not the case for systolic matrix array implementations of existing neural network models. Note that all PE elements across the entire matrix array perform the same translation and MAC operations in parallel. Further, in some embodiments, all PE elements execute the same code using a common clock signal (not shown), which implies that there is no need for a synchronization protocol when translating activation state vectors between PE elements.
[0098] [Fig. 10] schematically illustrates a translation sequence according to an expanding spiral between local memories of the matrix network of processing elements PE 1 to PE 9 of [Fig. 8] during a forward lateral mixing according to an exemplary embodiment of the present description. This translation sequence allows, for example, over a sequence of eight translation operations, the activation state vector of the central processing element PE 5 of the network to be communicated to each of its eight neighbors. As represented by arrows in [Fig. 10], the sequence begins at the element PE 5, and involves a translation sequence according to an expanding spiral L,U,R,R,D,D,L,L, where L is a translation to the left, U is an translation to the top, R is a translation to the right and D is a translation to the bottom.This means that the input section of the local memory 502 of the processing element PE 5 contains, over nine iterations, the activation state vectors of the processing elements: PE 5, PE 6, PE 9, PE 8, PE 7, PE 4, PE 1, PE 2 and PE 3. Of course, [Fig. 10] illustrates only one example of the translation sequence that can be applied, many alternative translation patterns existing, such as a frame sequence, in the case where the memory of each PE element includes a section that stores the hidden state vector of the PE element itself throughout the processing of the layer.
[0099] It will be noted that for some PE elements of the matrix network, they will receive activation state vectors that can be passed out of the matrix network before re-entering the matrix network. This is for example made possible by creating a margin at the outer boundary of the matrix network with dummy PEs that only perform the translation operation. Alternatively, in the case where the memory of each PE element includes a section that stores the hidden state vectors of the PE element itself throughout the layer processing, then the original vector is never lost and can be transferred.
[0100] In some embodiments, each of the PE elements of the matrix array of PE elements is configured to perform, synchronously with the other PE elements, a same sequence of operations, involving a translation of the neuron data of a neighboring PE element to the east, south, west, or north of the PE element depending on the translation sequence, to perform the same operation on the neuron data, and to export the neuron data by translation to a neighboring PE element to the east, south, west, or north of the PE element depending on the translation sequence. For example, this is achieved by the loading the same instruction code into the local memory of each PE element, and executing this same instruction code.
[0101] In the case where the DLM neural network has multiple hidden layers, each additional layer spatially increases the area of influence of a given macropixel, as will now be described with reference to [Fig.l 1].
[0102] Figure 11 schematically illustrates the radius of influence of a processing element PE2.2 of a matrix array of 6 by 6 PE elements after one, two and three layers of lateral mixing according to an example in which the weighting matrix has dimensions of 3 by 3 sub-matrices. It can be seen that, after the input layer 0 (AFTER LO), which does not include any lateral mixing in this example, the input data are present only in the processing element PEz,2. After the first hidden layer (AFTER L1), the area of influence covers a 3 by 3 group of PE elements centered on the processing element PE^, after the second hidden layer (AFTER L2) the area of influence covers a 5 by 5 group of PE elements centered on the processing element ^^2,2, and after the third hidden layer (AFTER L3) the area of influence covers the entire 6 by 6 PE element matrix array.
[0103] Thus, during each layer, each PE element will integrate information from increasingly distant sources into the matrix array of PE elements as described in [Fig. 11]. This allows each PE element to produce a local embedding, corresponding to an abstract representation of the input in the form of an output vector, which takes into account global characteristics and the context. In some embodiments, the number of lateral mixing layers is, for example, defined by the dimensions of the matrix array of PE elements and the number of layers that allow covering the complete image, which also depends on the resolution of each macropixel.
[0104] [Fig. 12] schematically illustrates an imaging device 1200 comprising an image sensor 1202 stacked on a data processing chip 1204, and in particular on a matrix array of PE elements 1205 of the processing chip 1204, according to an exemplary embodiment of the present description. The matrix array of PE elements 1205 is for example a systolic matrix array implementing the DLM neural network as described herein.
[0105] The image sensor 1202 comprises, for example, a matrix of pixel tiles (PIXEL TILE), each of which comprises, for example, a sub-matrix of c pixels (c PIXELS) by d pixels (d PIXELS), where c and d are equal to 16 in the example of [Fig. 12]. Each pixel comprises, for example, a photodetector. The image sensor 1202 also comprises, for example, analog-to-digital converters (not shown) for converting analog signals captured by each pixel into numerical values. merics which can be processed by the data processing chip 1204 by means of internal processing in the sensor.
[0106] Each pixel tile forms for example an upper tile and is positioned on a corresponding lower tile of the data processing chip forming a processing element (PE) of the matrix array of PE elements 1205, each pair of upper and lower tiles forming for example a 3D cluster (3D cluster). Each PE element comprises for example E calculation elements 1208, where E is equal to 16 in the example of [Fig. 12]. Each of the calculation elements processes for example a line of pixel data of a corresponding macropixel (MACROPIXEL) formed of p pixels of the pixel tile, where p is equal to 4 in the example of [Fig. 12].
[0107] Each pixel tile is for example coupled to the corresponding lower tile by interconnecting vias, also called 3D vias. Each lower tile further comprises for example the data / code memory 502, 504 and the local processor 508. In addition, the PE element comprises for example one or more additional processing circuits 1208 used for example to add parallelism in each PE element, allowing for example a row of pixels or activation vectors to be processed at the same time.
[0108] The processing chip 1204 also comprises, for example, attached to the matrix network of PE elements 1205, a data memory 1212, an instruction memory 1214, and a global control circuit 126 of the imaging device. The global control circuit 1216 is for example configured to provide a data stream 1218 from the imaging device 1200 comprising, for example, image data captured by the image sensor and inference data produced by the DML neural network implemented by the processing matrix.
[0109] [Fig. 13] is a flowchart illustrating steps of an inference method involving direct lateral mixing according to an exemplary embodiment of the present disclosure. The method of [Fig. 13] is for example implemented by a matrix array of processing elements as described herein, such as by the processing matrix 1205 of [Fig. 12], and by an image sensor.
[0110] At an operation 1301, a raw image is captured by an image sensor. The raw image may be a grayscale image or a color image. In some embodiments, such as that of [Fig. 12], the image sensor and the processing die are part of a single imaging device, while in other embodiments the image sensor may be a separate component. It is assumed that the image sensor captures grayscale images, in other words, raw pixel intensity, but the upper sensing layer could also include at the same time photodetectors sensitive to different wavelengths of light and produce color images composed of a number of color channels, such as RGB images. In addition, the raw image is converted by analog-to-digital converters in the image sensor, into a raw digital image.
[0111] At an operation 1302 following operation 1301, the digital pixel data of the raw image are loaded as macropixel input vectors into the matrix array of PE elements. For example, each PE element of the matrix array receives the pixel data from a corresponding macropixel. This data is for example stored in the input section of the local memory 502 of each PE element.
[0112] At an operation 1303 after operation 1302, each PE^j element of the matrix array of PE elements is configured to multiply the corresponding macropixel input vector by a corresponding weight matrix to produce an initial hidden state h^j. As described above, this input layer operation does not necessarily involve lateral mixing but serves, for example, essentially to reduce the size of the macropixel input vectors.
[0113] At an operation 1304 after operation 1303, the variables "n" and "CUMULATIVE RESULT" are for example each set to 0.
[0114] At an operation 1305 after operation 1304, for a subsequent layer L, each element PEij of the matrix array of PE elements is configured to multiply the hidden state by a corresponding weight matrix to produce a weighted product. For example, x and y are parameters shown in Figure 3 and at each iteration, the hidden state hi~y+yj+x of a different state among the nine hidden states to be processed by each PE element is processed. For example, at the first iteration, the hidden state of the PE element itself is processed.
[0115] At an operation 1306 after operation 1305, the weighted product generated at operation 1305 is added to the current cumulative result to produce a new cumulative result.
[0116] At an operation 1307 after operation 1306, it is determined whether n = 8, which is the number of iterations to be performed in the case where each weighting sub-matrix is of dimensions 3 by 3. Otherwise (branch N), at an operation 1308, n is incremented and an operation 1309 is performed. At operation 1309, a translation is for example performed to transfer to the PE element a next hidden state to be processed, which is a hidden state coming from one of the neighboring PEs. Operations 1305 to 1307 are then for example repeated in the manner described above, but starting from the hidden state imported by translation from the neighboring PE element. For example, according to the translation sequence of FIG. 10, after processing the hidden state of the PE element itself- Similarly, the hidden states of the eight neighboring PE elements are processed in the order hu+ij+b ^Lj+oj+b hm-u+b hu-ij-b h^+Qj-b hLj+ij_[.
[0117] When, at operation 1307, n = 8 (Y branch), this means that the cumulative result is equal to the sum aU-j of all weighted products (nine products in the present example), and at operation 1310, an activation function f() is applied to the cumulative result to produce a new hidden state h^j of the processing element
[0118] At an operation 1311 after operation 1310, it is determined whether the hidden layer being processed is the last one or whether there are additional hidden layers to be processed. If it is not the last one (branch N), for example, operations 1304 to 1311 are repeated for the last layer L from the new hidden state huj produced at operation 1310, and from a weighting matrix Wÿ associated with the new hidden layer, unless the same weighting matrix is to be applied.
[0119] When, at operation 1311, there are no more hidden layers to apply, at an operation 1312, an output layer is applied for example. This involves for example the multiplication, by each processing element PEî,j, of the hidden state huj resulting from the hidden layers by another weighting matrix. As for the input layer, it is possible to implement the output layer with or without lateral mixing. For example, if the task of interest is a binary segmentation for a region of interest (ROI) proposal, each PE element is for example configured to output either a single value (coarse segmentation by the PE elements) or a matrix of values which, for example, correspond to the size of a macropixel (fine segmentation by the PE elements).Assuming an application involving a coarse ROI proposal of PE elements, each PE element is for example configured to output a value between 0 and 1 that indicates the probability that this PE element is superimposed on a region of interest of the full field of view. Therefore, a weight matrix of 9xNxl or Nxl is for example applied by each PE element with and without lateral mixing respectively. Since, for this task, the weight matrix is relatively small, it may well be desired to perform lateral mixing in the output layer. In contrast, for a "fine ROI proposal for PE elements" in which, for an input macropixel resolution of 16x16, the weight matrix would be either 9xNx256 or Nx256, where 256 corresponds to the flattened square piece, it may be more economical not to perform lateral mixing in the output layer.For a task involving a coarse ROI proposal for PE elements, with output side mixing, the weight matrix occupies, for example, 72 bytes of memory. The full DLM, assuming an indi . individual for each layer and a total of 5 layers for a 16x12 PE element matrix array, uses for example 1.2 MB of memory across the entire PE element matrix array. If shared hidden layer weights are used, this drops to 230 kB for all PE elements.
[0120] An example of the complete algorithm is summarized in pseudo-code in Algorithm 1 below. This example relies on the particular spiral translation strategy described in relation to [Fig. 10], but other translation strategies could be used. Algorithm 1 describes in particular the operations that are executed simultaneously on all PE elements, where w_in is the weighting matrix applied to the input layer, B_in is a bias to be applied at the input layer, which can be zero, w_h is the weighting matrix applied to a current hidden layer comprising the sub-matrices w_h[0] to w_h[8], B_h is a bias to be applied to the current hidden layer and h[0] is the hidden state of the PE element itself, and h[l] to h[8] are the hidden states of the eight neighboring PE elements.Indeed, each layer can use a bias parameter such that, instead of being initialized as a vector of zeros for each layer, the cumulative weighted sum of each hidden layer neuron is initialized with a value that has been learned.
[0121] Algorithm 1 - An inference using a DLM
[0122] Capture input macropixel X
[0123] h = X*w_in + B_in
[0124] For each DLM layer
[0125] If weight not shared between layers
[0126] Load w_h, B_h from local memory
[0127] If using a bias
[0128] = B_h
[0129] Otherwise
[0130] = zeros
[0131] = O + h*w_h[0]
[0132] Translation of h 1 to the LEFT
[0133] = O + h*w_h[l]
[0134] Translation of h 1 UP
[0135] = O + h*w_h[2]
[0136] Translation of h 1 to the RIGHT
[0137] = O + h*w_h[3]
[0138] Translation of h 1 to the RIGHT
[0139] = O + h*w_h[4]
[0140] Translation of h 1 DOWN
[0141] = O + h*w_h[5]
[0142] Translation of h 1 towards DOWN
[0143] = O + h*w_h[6]
[0144] Translation of 1 to the LEFT
[0145] = O + h*w_h[7]
[0146] Translation of 1 to the LEFT
[0147] = O + h*w_h[8]
[0148] h = f(O)
[0149] As known to those skilled in the art, the weight matrices of a neural network can be learned using a standard back-propagation algorithm and this also applies to the weight matrices of the DLM neural network described herein. Furthermore, if desired, bias values as described above can also be learned.
[0150] In some cases, the back-propagation algorithm itself may also use lateral mixing, as will now be described in more detail with reference to Figures 14 to 18.
[0151] [Fig. 14] schematically illustrates a rearward lateral mixing using a matrix array of processing elements according to an exemplary embodiment of the present disclosure.
[0152] [Fig. 14] illustrates the same DLM neural network layers 400 to 403 as [Fig. 4], but illustrates how lateral mixing can be applied to on-chip learning. Conceptually, this is equivalent to calculating the gradient of a loss (LOSS) at or after the output layer by aggregation, and reversing the operations that were performed so that the gradient flows back through all intermediate PE elements to the point where the initial data was fed into the matrix network of PE elements.
[0153] In Figure 14, the arrows indicate the flow of information returning through the lateral mixing layers from the output to the input. This involves, for each layer from the output layer L to the input layer L-3, the calculation of a corresponding error signal ÔL to ÔL^ as will now be described with reference to [Fig. 15].
[0154] Figure 15 schematically illustrates a back-propagation operation according to an exemplary embodiment of the present description in the case of a simple neural network having two hidden layers L-2 and Ll and the output layer L. During forward propagation, the weighted sums of the inputs of each layer are designated by aL-2, aL-i and ai, the hidden states resulting from each layer are designated by hL_2, and hL, the weighting matrices applied to the outputs of layers Ll and L are designated by VF and WL, and the activation function applied to the neuron in the output layer to produce the output state hL is
[0155] A loss associated with the output state hL is denoted by C(hL, t}- The gradient for a weight layer in a layer 1 of the network is: [Math 3] where 5) is an error signal that comes either from the loss in the output layer or from the next layer in front for all other layers.
[0156] In the output layer L, the value of delta is: [Math 4] and for all other layers, the delta value is: [Math 5] ^4-1 = ¾ * / ^(^-1) where ° is the Hadamard product or by elements.
[0157] [Fig. 16] is a flowchart illustrating steps of a method for training a DLM neural network, according to an exemplary embodiment of the present invention. This method involves backward lateral mixing. The method of [Fig. 16] is for example implemented by a matrix network of processing elements as described herein, such as by the processing matrix 1205 of [Fig. 12]. [Fig. 16] will also be described with reference to [Fig. 17].
[0158] [Fig. 17] schematically illustrates back-propagation operations of several layers L, L1 and L-2 in a matrix network of PE elements according to an exemplary embodiment of the present description.
[0159] As shown in [Fig. 16], at an operation 1601, the value 8L is calculated for each PE element of the matrix network. For example, this value is calculated using the following equation: [Math 6] cL = £>,4) * / -(¾) where C ( hL. t ) is the derivative of the loss C(h^ t)-
[0160] As shown in Figure 17, the value for a given PE element is based on the loss t) associated with this PE element, which is based on the activation state hL.
[0161] Also in operation 1601, a variable n is for example set equal to 0.
[0162] At an operation 1602 after operation 1601, the local weight gradients are calculated for each PE element. For a given PE element, this involves calculating the local weight gradients for each of the nine sub-matrices of the element PE given using the activation states from the L1 layer. For example, each gradient is calculated for a PE element from the following equation: [Math 7] for x G [ - 1,0, 1] and y G [ - 1,0, 1] .
[0163] This uses the activation states of the neighbors and on the PE element itself from the previous layer, which are either still stored locally from the forward pass in each PE element or alternatively loaded from an external memory, for example depending on the memory capacity of the PE element. For example, as shown in Figure 17, the activation y3 is used at the processing element during one iteration. The activation states of neighboring PE elements of a given PE element are for example provided to this PE element during a number of iterations, as described in connection with Figures 5 to 10 above, involving operations 1603, 1604, 1605 and a repetition of operation 1602.
[0164] At operation 1603, for example, it is determined whether the parameter n is equal to 8, which is the number of neighboring PE elements from which the activation state must be received. If this is not the case (branch N), at operation 1604, n is incremented, then at operation 1605, a translation is performed to import by translation the activation state from one of the eight neighbors. Operation 1602 is then repeated from the new activation state and operations 1603 to 1605 are repeated. This loop is repeated until the activation states of all neighboring PE elements, as well as of the PE element itself, have been processed and used to generate the corresponding gradients.
[0165] Following the generation of each gradient at operation 1602, the corresponding weight in the weighting matrix is for example updated, for example from the following equation: [Math 8] where 7 is a learning rate. In some embodiments, this involves loading the weight sub-matrices of each of the layers from on-board memory, such as memory 1212 of [Fig. 12].
[0166] Once na reaches 8 at operation 1603, at a subsequent operation 1606, each PE element is configured to generate in each PE element a phi matrix, ^>{ Lj3,\k} , which is a set of nine vectors, for example a matrix of vectors 3x3 - one for each of the weighting sub-matrices on each PE element for each layer. Phi corresponds to an error signal that is used by the neighboring PE element to calculate the delta signal in the previous layer. Phi is for example calculated from the following equation:
[0167] [Math.9]
[0168] As shown in Figure 17, the parameters [Math.9] , are calculated locally for a current layer [Math.9] L then used in the layer [Math.9] Ll to calculate the delta vector for each PE element. Figure 17 illustrates an example for the parameter [Math.9] for the PE element at i = 2, j = 2, which is used in the layer [Math.9] L- 1 by the PE element i = 3, j = 3.
[0169] After operation 1606, the processing moves to the next layer and at an operation 1607, the parameter n is for example again set to zero.
[0170] After operation 1607, a series of operations 1608, 1609, 1610 and 1611 is repeated by each PE element of the matrix array, for several iterations to generate the value. For example, the value is calculated using the following equation:
[0171] [Math. 10] * i
[0172] This involves the addition of a set of nine different variables from [Math. 10] from the neighboring PE elements determined at operation 1606 of the next layer in the weighting matrix for PE elements i,j with each of its eight neighbors in i and j.
[0173] The values of from the neighboring PE elements of a given PE element that are used to generate j are for example imported into this PE element by translation during a certain number of iterations, as described in relation to Figures 5 to 10 above. For example, at operation 1609, it is determined whether the parameter n is equal to 8, which is the number of neighboring PE elements from which the values must be received. If this is not the case (branch N), at operation 1610, n is incremented, then at an operation 1611, a translation is carried out to import by translation the values of P from one of the eight neighbors. Operation 1608 is then repeated from the new values of V to partially generate a new part of the value 8^ this is accumulated with the partial value(s), then operations 1609 to 1611 are repeated.This loop is repeated until the values from all neighboring PE elements have been processed and the cumulative result is equal to ÔLA. .
[0174] Once na reaches 8 at operation 1609, at a subsequent operation 1612, the variable n is for example set to zero again.
[0175] At an operation 1613 after operation 1609, the local weight gradients are calculated for each PE element. For a given PE element, this involves calculating the local weight gradients for each of the nine sub-matrices of the given PE element using the activation states from the LL layer. For example, each gradient is calculated from the following equation:
[0176] [Math. 11] of
[0177] As in operation 1602, this uses the activation states of the neighbors, which are always locally stored by each PE element. For example, as shown in Figure 17, the activation [Math. 11] ^ / .-2.44 is used by the processing element [Math. 11] pe33 of the LL layer The activation states of the PE elements neighboring a given PE element are for example provided to this PE element during a certain number of iterations, as described in relation to Figures 5 to 10 above, involving operations 1614, 1615, 1616 and a repetition of operation 1613.
[0178] In operation 1614, we determine for example whether the parameter n is equal to 8, which is the number of neighboring PE elements from which the activation state is to be received. If this is not the case (branch N), at operation 1615 n is incremented, then at operation 1616 a translation is performed to import by translation the activation state from one of the eight neighbors. Operation 1613 is then repeated from the new activation state and operations 1614 to 1616 are repeated. This loop is repeated until the activation states of all neighboring PE elements have been processed and used to generate the corresponding gradients.
[0179] Following the generation of each gradient in operation 1613, the corresponding weight in the weighting matrix is for example updated as described in relation to operation 1602.
[0180] Once na reaches 8 at operation 1614, at a subsequent operation 1617, each PE element is configured to generate in each PE element the matrix phi which is a set of nine vectors, for example a 3x3 matrix of vectors - one for each of the weighting sub-matrices on each PE element for each layer. Phi is for example calculated from the following equation: [Math 12]
[0181] Figure 17 illustrates an example for the parameter ^-1,4,4,ti, for the PE element at i = 3, j = 3, which is used in the L - 2 layer by the PE element i = 4, j = 4.
[0182] At an operation 1618 following operation 1617, it is determined for example whether the current layer L is the input layer. If this is not the case (branch N), at an operation 1619, the variable n is set to zero, then operations 1608 to 1617 are repeated from the previous layer L.
[0183] When, at operation 1618, it is determined that the current layer L is the input layer (branch Y), the method ends for example at an operation 1611.
[0184] [Fig. 18] is a graph representing test results according to an exemplary embodiment of the present disclosure. In particular, for a coarse multi-class segmentation task for PE elements, it was investigated how hidden layers affected the performance of the DLM neural network, and [Fig. 18] illustrates the intersection over union test result (TEST loU) as a function of the epoch of the test data (EPOCH). It can be seen that, although they share the same hidden weights, the performance drop is relatively small, and is for example tolerable for at least some applications given the gain in execution efficiency by not having to load new weight matrices for each layer.
[0185] An advantage of the embodiments described herein is that the neural network implementation using a matrix network of PE elements can have relatively low latency, in part because at least some of the data to be processed in relation with each layer of the network remain within or near the PE element that will use it, and as a result relatively few data transfers are required. In addition, this also allows for a reduction in memory requirements and circuit power consumption.
[0186] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these embodiments may be combined and other variations will occur to those skilled in the art. For example, while embodiments have been described in which a MAC operation is applied after each forward lateral shuffle iteration such that only the cumulative result is stored after each iteration, in alternative embodiments, each forward lateral shuffle operation could involve only storing in local memory the hidden vectors of neighboring PE elements. The hidden vectors are then, for example, concatenated together and the MAC operation is, for example, applied only once on a larger matrix. This variation involves using a larger local memory 502 in each PE element to store all of the hidden layer data.
[0187] Furthermore, while embodiments based on a 2D matrix array of PE elements have been described, it will be apparent to those skilled in the art that the principles described herein could also be applied to a 3D matrix array of PE elements, where a 3D embedding process is for example used to embed multiple layers of matrix arrays of PE elements on top of each other. In this case, the data is translated not only between one and its 8 neighbors in a square neighborhood, but between it and 26 neighbors in the upper and lower layers in a cubic volume neighborhood.
[0188] In the described embodiments, the PE elements exchange hidden state vectors only with immediate neighbors. However, the PE elements could alternatively or otherwise perform lateral mixing with the PE elements of the matrix array that are spaced apart by a step of 2. In such a case, it would be possible to use the same expanding spiral pattern as described in connection with [Fig. 10], in which the first data translation operations translate the data from the corresponding "ring", and subsequently the data can be translated one step at a time.
[0189] Furthermore, the hidden layer vectors can be of different sizes for two different layers of the network. This means that for each layer, the weight matrix that is loaded from the external memory does not always have the same dimensions, for example, and the local sections of the PE memory can in this case be reformatted between the layers.
[0190] In addition, residual connections can also be added to the DLM model, as described in the publication by He, Kaiming et al. entitled "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. For this, a fourth memory section is for example added to the local memory 502 of each PE element which is configured to store the hidden vector of the previous layers. After updating the hidden vector in a DLM layer, the results are then for example added to these previously stored hidden vectors. An advantage of this approach is that the information from the previous layers is preserved for deeper models.
[0191] In the described embodiments, the main operation applied to the hidden vector states is a fully connected layer, sometimes called a linear layer. This corresponds to the calculation of the inner product between an input vector and a weight matrix. However, many variations are possible when this main operation changes. For example, the following two variations could be applied.
[0192] According to the first variant, in a 2D convolutional deep lateral mixer, each PE element stores convolution kernels and the data imported by translation from neighboring PEs are different channels of an input feature map. The convolution kernels are traversed over the input feature maps to produce hidden state feature maps, which will then be used in the next layer as input feature maps. For example, if each PE element contains a 1-channel feature map, it will operate on an effective 9-channel input feature map.To allow the number and size of feature maps to be constant across DLM layers, nine convolution kernels learned with stride 1 and margin 1 can be applied to this input, resulting in a local hidden-state feature map of 1 channel (after summing the resulting partial feature maps from nine channels). Alternatively, for example at least for the input layer, a stride larger than one can be used to reduce the size of hidden feature maps in successive layers. Since a single feature map would require less memory, one can then store hidden feature maps with a larger number of channels.
[0193] According to the second variant, a DLM transformer is implemented, in which an attention mechanism can be used, for example in the form of another weighting matrix that delivers attention weights, to weight certain neighboring PE elements and certain neurons with higher or lower importance. The concept of applying transformers in computer vision tasks is described in more detail in the publication by Dosovitskiy et al. Entitled "An image is worth 16x16 words: Transformers for image recognition at scale", arXiv preprint arXiv:2010.11929 (2020).
[0194] Furthermore, the input data could be an event camera instead of an RGB or black and white / grayscale imager. In this case, the input pixels are, for example, binary events, based on changes in light intensity at the output.
[0195] DLMs can also be useful for performing an internal sensor compression of the captured scene that can be sent to another, perhaps more powerful, computing system for further processing, and which will use this compressed representation for a downstream task or even possibly to reconstruct the complete image. Such compression can be learned by training a so-called autoencoder model. An autoencoder is composed of two parts - the encoder that reduces the input by projection to a low-dimensional representation - and a decoder that uses this representation to reconstruct the original input image. The DLM, as described above, can be used as an encoder in which each PE element outputs a vector of a given size, for example a 4-element vector. The complete output on the matrix array of PE elements corresponds to a 16x12x4 matrix array.This can be interpreted as a 4-channel 16x12 feature map. This embedding can then be input to a second model, such as a convolutional decoder, which expands the input over a series of further convolutional layers using 2D transposed convolutions. The output of this model is an image with the same dimensions as the input scene captured on the PE array. To train the parameters of the DLM and CNN, for example, a pixel-wise mean squared error loss can be calculated and back-propagated through both models. After training, the decoder is for example able to reconstruct images from the compressed representation output by the DLM. The DLM effectively learns how to abstract and compress scene information.For example, it has been found that a digital input structure and location can be reconstructed very faithfully with a relatively high compression rate of 98.4%. In such an application, PE elements are for example used to perform sensor-internal pre-processing of the data image before sending the compressed representation to a downstream task, such as localizing or classifying an object in the scene, which has the advantage of providing a significant amplification of the available system bandwidth.
[0196] Applications of the embodiments described herein include, for example, image segmentation and object detection. In the case of object detection, instead of outputting, for each class, a single value or vector of values per pixel or macropixel, the PE element of each pixel or macropixel is configured to output a vector containing the coordinates of one or more boundary boxes, the class distribution for each of these boundary boxes and, optionally, an "objectness" score (assessment of the quality of "object"). Techniques such as non-max suppression are for example used for an implementation on the matrix network of PE elements to downsample the boxes. Typically, this method keeps the boxes with the "objectness" score of a series of boxes that have a high intersection over union with each other.
[0197] Another application of the embodiments described herein is a region proposal network. According to this application, an implementation of the DLM such as a sensor-integrated 3D processing solution enables efficient means of proposing regions of interest to be processed by a subsequent convolutional neural network model. The convolutional neural network is for example configured to operate on the output image piece identified by the DLM and the result of the approach is therefore equivalent to a detection. For example, the first processing phase is performed on a faster RCNN model, as described in the publication by Ren, Shaoqing, et al., entitled "Faster r-cnn: Towards real-time object detection with region proposai networks", Advances in neural information processing Systems 28 (2015).This is a particularly advantageous application since the region of interest detection phase of such approaches constitutes the bottleneck and the most time-consuming and resource-intensive part.
[0198] Furthermore, as already mentioned, applications of the embodiments described herein include applications in which the input data are not image data. For example, while embodiments based on a 2D matrix array of PE elements have been described, in an alternative embodiment, the principle described herein could be applied to a 1D matrix array of PE elements, in which each PE element has two neighbors, except for the PE element at the ends of the matrix array. Such a 1D matrix array is, for example, suitable for processing time-series data. For example, many sensing systems, such as EEG headsets and neural interfaces, for example, are composed of a matrix array of analog time-series signals. The PE element may therefore be integrated below a matrix array forming an interface for such signals to the PE elements.Values from the time series can be sampled at a given rate, for example in a manner similar to how pixels are transduced from photons at a given rate, and DLM can be applied using a systolic matrix array of PE elements in a manner similar to that done for images.
[0199] Finally, the practical implementation of the embodiments and variants described herein is within the reach of those skilled in the art from the functional description provided above.
Claims
Claims
1. An electronic circuit implementing a neural network, the electronic circuit comprising: - a matrix array (200, 500) of processing elements (PE), each processing element (PE) comprising a data processing circuit (508) and a local memory (502), each processing element (PE) implementing one or more neurons of the neural network, the local memory (502) of each processing element (PE) being configured to store neuron data (hi,yj, associated with the one or more neurons implemented by the neural network; and - a set of data propagation circuits (503, MUX_IN, MUX_OUT) configured to perform a forward or backward lateral mixing of the neuron data stored by the local memory (502) of each processing element by the propagation, performed synchronously by each processing element,neuron data to the local memory (502) of each processing element from the local memory (502) of one or more neighboring processing elements of each processing element during a plurality of processing iterations, wherein each of the processing elements is configured to receive, synchronously at each iteration, the neuron data stored by the local memory of one of the plurality of neighboring processing elements, at least partially overwriting the neuron data stored in its local memory, to process, during each processing iteration, the received neuron data to produce updated neuron data (hu([h . ) and to store the updated neuron data (h^j, (pL.. in its local memory (502), wherein each processing element is configured to use the updated neuron data ( h^j, <pL fj v v) lors d’une itération de traitement ultérieure.,
2. The electronic circuit of claim 1, wherein the data processing circuit (508) of each processing element (PE) is further configured to calculate a partial intermediate value) from neuron data from the one or more neighboring processing elements.
3. An electronic circuit according to claim 2, wherein the network neural network comprises an input layer and at least two hidden layers, the input layer and the at least two hidden layers being implemented by the matrix network of processing elements, wherein the data processing circuit (508) of each processing element (PE) is configured to calculate the partial intermediate value) from the neuron data in relation to each of the hidden layers.
4. An electronic circuit according to any one of claims 1 to 3, wherein the data propagation circuitry (503, MUX_IN, MUX_OUT) is configured to propagate the neuron data to the local memory (508) of the one or more neighboring processing elements of the matrix network according to a translation sequence following an expanding spiral.
5. An electronic circuit according to any one of claims 1 to 4, wherein the data propagation circuitry (503, MUX_IN, MUX_OUT) comprises, for each processing element (PE), at least one multiplexer (MUX_IN, MUX_OUT) configured to choose from which neighboring processing element and / or to which neighboring processing element the neuron data is propagated.
6. An electronic circuit according to any one of claims 1 to 5, wherein the matrix array (200, 500) of processing elements (PE) is configured to receive one or more images as input data.
7. An electronic circuit according to any one of claims 1 to 6, wherein the data propagation circuitry (503, MUX_IN, MUX_0UT) is configured to perform forward lateral mixing of the neuron data, and wherein the neuron data of each processing element (PE) is an activation state (of the one or more neurons implemented by the processing element (PE).
8. An electronic circuit according to any one of claims 1 to 6, wherein the data propagation circuitry (503, MUX_IN, MUX_0UT) is configured to perform a lateral backward mixing of the neuron data, wherein the neuron data of each processing element (PE) comprises: - an activation state (h^j) of the one or more neurons implemented by the processing element (PE); and / or
9.
10.
11. - a parameter (^ j representing an error signal. An electronic circuit according to any one of claims 1 to 8, wherein the neural network is a convolutional neural network. An imaging device comprising a stack formed of an upper level comprising an image sensor (1202) and a lower level (1204) comprising the electronic circuit according to any one of claims 1 to 9. Method for storing, propagating and processing data in a neural network, this method comprising: - the storage of neuron data) associated with one or several neurons implemented by the neural network in a local memory (502) of each processing element (PE) of a matrix network (200, 500) of processing elements (PE), each processing element (PE) of the matrix network comprising a data processing circuit (508) and the local memory (502); - the calculation, by the data processing circuit (508) of each processing element (PE), of an intermediate value from the neuron data; - performing, by a set of data propagation circuits (503, MUX_IN, MUX_0UT), a lateral forward or backward mixing of the neuron data stored in the local memory (502) of each processing element by propagating, performed synchronously by each processing element, the neuron data to the local memory (502) of each processing element from the local memory (502) of one or more neighboring processing elements of each processing element during several processing iterations, wherein each of the processing elements is configured to receive, synchronously at each iteration, the neuron data stored by the local memory of one of the several neighboring processing elements, at least partially overwriting the neuron data stored in its local memory, to process, during each processing iteration,the received neuron data to produce updated neuron data (h-mj, <PL jj y y) et pour stocker les données de neurone mises à jour (hmj, (pLj. v y) dans sa mémoire locale (502) ; et, - the use of updated neuron data (hu(pT . ., .) during of a subsequent processing iteration.
12. The method of claim 11, further comprising calculating, by the data processing circuit (508) of each processing element (PE), an intermediate value Ô^yj partial) from neuron data from the one or more neighboring processing elements.
13. The method of claim 12, wherein the neural network comprises an input layer and at least two hidden layers, the input layer and the at least two hidden layers being implemented by the matrix network of processing elements, the method further comprising calculating, by the data processing circuit (508) of each processing element, the intermediate value (^^, ôyyj partial) from the neuron data in relation to each of the hidden layers.
