Multifunctional electronic component, corresponding device, methods and programs

Abstract

The invention relates to an electronic data processing platform comprising a general processor (PS) and a "level 1" memory (L1). This platform further comprises a programmable logic component (PL) comprising a memory referred to as "level 0" (L0), the programmable logic component being parameterized so as to define typologies of executable calculations, said platform further comprising orchestration software running on the general processor, the orchestration software being responsible for providing, to the programmable logic component, a set of configuration data for the running thereof, within the limits of the typologies of executable calculations.

Description

[0001] DESCRIPTION

[0002] TITLE: Multifunctional electronic component, device, methods and corresponding programs.

[0003] Technical field

[0004] The invention relates to the implementation of multifunctional electronic components. The invention relates more particularly to the implementation of multifunctional electronic components that can be reconfigured on the fly according to needs. The invention more naturally finds application in the implementation of neural networks on embedded architectures.

[0005] Prior art

[0006] An artificial neural network (or ANN) can be thought of as a complex system combining and interconnecting a large number of non-linear processing units called neurons. Some artificial intelligence algorithms are well known for supervised learning, in which an artificial neural network is used to make a prediction from an input layer receiving input data to an output layer, as shown in Figure 1. There are many types and variations of neural networks, including for example the following three major types:

[0007] The multilayer perceptron (or MLP) is the extended version of the original perceptron that contains three or more layers. These layers are also called fully connected layers (or FCs), and although they are not used as stand-alone entities, they are usually integrated as subcomponents of other architectures (such as in CNNs).

[0008] Convolutional neural networks (or CNNs) are neural networks specialized primarily in image processing applications. They are based on convolution operations (matrix operations) commonly used in image processing (such as the Sobel filter).

[0009] Recurrent neural networks (RNNs) contain internal memory entities that enable a feedback loop. This principle allows for a temporal relationship to be taken into account in the processed data. Therefore, the output of a cell is not only influenced by its input but also by previous computations, which allows for capturing a certain temporal relationship inherent in the data.

[0010] The inventors identified that current paradigms for implementing and training neural network artificial intelligence algorithms are mainly based on centralized architectures with heavy software infrastructures such as server farms, usually GPU-based and even containing some FPGA / ASIC computing accelerators. The training of execution parameters is done online and once trained, the network (and its parameters) is transferred to an inference target for a dedicated task or mission.

[0011] Based on this analysis, the inventors found that: a) Artificial intelligence algorithms (especially those comprising large networks) are difficult to transpose onto highly constrained embedded systems in terms of size, energy consumption and real-time constraints. b) All current embedded approaches are based on online learning - offline execution explained above. The training of neural networks (i.e. obtaining its operating parameters), which is the part that consumes the most time, memory and computing resources, is not performed directly on the embedded systems. The networks are trained online, their data is stored, and then they are implemented as is on the destination electronic board.c) Once a neural network has been designed, trained and optimally integrated into an embedded device, it is difficult to modify it (for another task, for a new configuration, a more efficient architecture, ...). Especially when using FPGAs, this reconfiguration / re-implementation of the neural network can be a long and complex task.

[0012] In other words, the inventors have found that implementing a neural network in an embedded architecture suffers from many problems, the first of which is the fixed nature of the network thus implemented. It is certainly possible, even in an FPGA, to modify the weights of the neural network, in order to take into account improvements in learning (performed online), but it is difficult to envisage modifying the functionalities of the neural network thus obtained. Among the non-modifiable functionalities, enlarging the size of the neural network poses often insoluble problems. In addition, embedded neural networks suffer from limitations in learning, which is incompatible with the low power consumption requirements and the limited computing power of these embedded systems.

[0013] There is therefore a need to provide data processing solutions for neural networks on embedded architectures, which can resolve the problems posed by the prior art.

[0014] Summary of the invention

[0015] The present technique makes it possible to propose a solution aimed at remedying certain drawbacks of the prior art.

[0016] According to a first aspect, the invention describes the manner in which the inventors decided to formalize the operations carried out within the framework of a series of calculations. More particularly, the invention relates to the serialization and parallelization of calculations carried out on input data to produce output data. The series of calculations in question may be the implementation of one or more neural networks, or any other digital calculations carried out from input data to produce output data.

[0017] According to a second aspect, from the modeling of the operations carried out within the framework of a series of calculations, the inventors have defined a methodology for creating electronic components capable of implementing the series of calculations in question in a generic manner.

[0018] According to a third aspect, once the electronic component is materialized (i.e. for example created or melted), the invention relates to a method of configuring and using this electronic component in order to execute one or more series of calculations modeled according to the methodology defined by the inventors.

[0019] Thus, the present technique relates in particular to an electronic data processing platform, comprising a general-purpose processor and a so-called level 1 memory, said platform further comprising a programmable logic component comprising a so-called level 0 memory, the programmable logic component being configured so as to define typologies of executable calculations. Said platform further comprises orchestration software executing on the general-purpose processor, the orchestration software being responsible for providing, to the programmable logic component, a set of configuration data for its execution within the limits of the typologies of executable calculations.

[0020] In a particular embodiment, the programmable logic component is an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0021] In a particular embodiment, the programmable logic component is configured to perform inference calculations for artificial intelligence.

[0022] In a particular embodiment, the programmable logic component is configured to perform learning calculations for artificial intelligence.

[0023] In a particular embodiment, the orchestration software is arranged to provide the execution component with a level 1 memory address at which the configuration data set is accessible, and to provide the programmable logic component with an execution command.

[0024] In a particular embodiment, the programmable logic component is further configured to extract the data structures and functional characteristics of the calculations to be executed and configured to determine how to read and write data in level 1 (L1) memory and determine an execution logic to be implemented.

[0025] In a particular embodiment, the execution logic to be implemented belongs to the group comprising: a multi-layer perceptron type neural network; a convolutional type neural network; a recurrent neural network.

[0026] Such a list is given for illustrative and non-limiting purposes, and the execution logic can be associated with other types of neural networks or learning (including reinforcement learning approaches for example).

[0027] In a particular embodiment, the programmable logic component is configured to copy, within its level 0 memory, said configuration data set, and to perform the calculations required by the configuration data of the configuration data set.

[0028] In a particular embodiment, the programmable logic component is configured to recursively load, into the level 0 memory, input data from the level 1 memory, then to calculate, from said input data, output data and to copy said output data into the level 1 memory.The proposed technique also aims at a method for obtaining an electronic platform as described previously in any of its different embodiments, in which the creation of the programmable logic component comprises the following steps: obtaining a parameterization file of the programmable logic component, comprising, for at least one characteristic of said programmable logic component, a value representative of the capacity of the programmable logic component to have this characteristic; compiling the parameterization file of the programmable logic component, delivering on the one hand a binary file for implementing the programmable logic component, and on the other hand an executable file for orchestrating the operation of the programmable logic component; creating the programmable logic component according to said binary file; loading said executable file by the general processor.

[0029] According to another aspect, the proposed technique also aims at a method of using the programmable logic component of an electronic platform as described previously in any one of its different embodiments, this method comprising a step of configuring the programmable logic component using configuration parameters, and at least one iteration of the following steps: loading a set of execution parameters; loading a set of execution variables; calculating at least one execution result as a function of the set of execution parameters and execution variables; recording, within a memory area of ​​the programmable logic component, said at least one execution result.

[0030] Accordingly, the present technique also relates to programs, capable of being executed by a computer or by a data processor, these programs comprising instructions for controlling the execution of the steps of the methods as mentioned above.

[0031] A program may use any programming language, and may be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0032] The present technique also aims at an information medium readable by a data processor, and comprising instructions of a program as mentioned above.

[0033] The information carrier may be any entity or terminal capable of storing the program. For example, the carrier may include a storage medium, such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or a magnetic recording medium, for example a mobile medium (memory card) or a hard disk or an SSD.

[0034] Furthermore, the information carrier may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means. The program according to the present technique may in particular be downloaded over a network such as the Internet.

[0035] Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to carry out or to be used in carrying out the method in question.

[0036] According to one embodiment, the present technique is implemented by means of software and / or hardware components. In this regard, the term "module" may correspond in this document to a software component, a hardware component or a set of hardware and software components.

[0037] A software component corresponds to one or more computer programs, one or more sub-programs of a program, or more generally to any element of a program or software capable of implementing a function or a set of functions, as described below for the module concerned. Such a software component is executed by a data processor of a physical entity (terminal, server, gateway, set-top-box, router, etc.) and is likely to access the hardware resources of this physical entity (memories, recording media, communication buses, electronic input / output cards, user interfaces, etc.).

[0038] Similarly, a hardware component is any element of a hardware assembly capable of implementing a function or set of functions, as described below for the module concerned. It may be a programmable hardware component or one with an integrated processor for running software, for example an integrated circuit, a smart card, a memory card, an electronic card for running firmware, etc.

[0039] Each component of the system described above of course implements its own software modules.

[0040] The different embodiments mentioned above can be combined with each other for the implementation of the present technique. Figures

[0041] Other aims, characteristics and advantages of the invention will appear more clearly on reading the following description, given as a simple illustrative, and non-limiting, example, in relation to the figures, among which:

[0042] [Fig 1] illustrates the general principle of a method for creating a programmable logic component, in a particular embodiment of the proposed technique;

[0043] [Fig 2] illustrates the general principle of a method of using a programmable logic component, in a particular embodiment of the proposed technique;

[0044] [Fig 3] presents an example of a configuration file for a programmable logic component, in a particular embodiment of the proposed technique;

[0045] [Fig 4] schematically describes the result of a parameterization operation of a programmable logic component, in a particular embodiment of the proposed technique;

[0046] [Fig 5] describes an example architecture of a data processing platform in a particular embodiment of the proposed technique;

[0047] [Fig 5a] describes another example of architecture of a data processing platform, in another particular embodiment of the proposed technique;

[0048] [Fig 6] shows examples of interfaces of a programmable logic component having drive capabilities, in a particular embodiment of the proposed technique;

[0049] [Fig 7] presents examples of interfaces of a programmable logic component having inference capabilities, in a particular embodiment of the proposed technique;

[0050] [Fig 8] describes an example of interactions between Manager and Executor, in a particular embodiment of the proposed technique;

[0051] [Fig 9] illustrates an example of using a programmable logic component for implementing a multi-layer MLP neural network, in a particular embodiment of the proposed technique;

[0052] [Fig 10] illustrates an example of use of a programmable logic component for the implementation of a Lenet-5 type CNN neural network, in a particular embodiment of the proposed technique;

[0053] [Fig 11] illustrates an example of use of a programmable logic component for the implementation of an LSTM type RNN neural network, in a particular embodiment of the proposed technique;

[0054] [Fig 12] describes the steps of calculating a first output data element by a programmable logic component, in a particular embodiment of the proposed technique; [Fig 13] describes the steps of calculating a second output data element by a programmable logic component, in a particular embodiment of the proposed technique.

[0055] Detailed description of the invention

[0056] 1. Reminders of the principles

[0057] As explained above, the invention aims to provide a computing platform, in particular for the execution of artificial intelligence algorithms, which is versatile and energy-efficient. To do this, the inventors have implemented a method for creating a pseudo-generic computing component, this component being associated, within a platform, with a general-purpose processor, to provide a flexible solution for executing artificial intelligence algorithms, in particular. The pseudo-generic component is the result of work to generalize calculations, in particular those carried out in the case of artificial intelligence algorithms, work which is presented subsequently.The general principle of the invention consists in designing a pseudo-generic component (also called a programmable logic component or execution component), this "pseudo-generic" design consisting in defining the calculations that this execution component is capable of performing. Once the pseudo-generic component has been parameterized (i.e. once the types of calculations that this component can perform have been delimited), this execution component is configured on the fly (i.e. at the time when its use is required) to make it perform specific calculations. The on-the-fly configuration of the component consists of transmitting a configuration file. This configuration file includes the parameters (functions, values, coefficients, etc.) necessary for implementing the calculations, within the genericity limits determined at the time of design.The organization of the implementation of the pseudo-generic component is carried out by a general-purpose processor, associated, within an execution platform, with this pseudo-generic component. This platform, overall, can be presented in the form of a SOC (from the English for "system on chip"). The execution platform thus includes a general-purpose processor, having the capacity to access a so-called level 1 (L1) memory. The general-purpose processor also has access to one or more external interfaces, such as network interfaces, USB, UART, SPI, Ethernet ETH, etc. The general-purpose processor executes software in charge of orchestrating the executions of the pseudo-generic component. It is this software which provides the configuration file "on the fly" and data on which the calculations must be carried out.Once this configuration and data are received, the pseudo-generic component (or programmable logic component) performs the calculations requested by the configuration file. To do this, the programmable logic component jointly implements its own memory (called level 0 memory, or LO) and the platform's level 1 memory. Thus, the programmable logic component also has instructions allowing read and write access to this level 1 memory.

[0058] Thus, in general, on the basis of the generalization of the calculations carried out by the inventors, the method for creating the programmable logic component which is the subject of the invention comprises, as illustrated in Figure 1 in a particular embodiment:

[0059] A step 11 of obtaining a parameterization file of the programmable logic component, comprising, for at least one characteristic of said programmable logic component, a value representative of the capacity of the programmable logic component to have this characteristic;

[0060] A compilation step 12 of the parameterization file of the programmable logic component, delivering, on the one hand, a binary file (of the “bitstream” type) for implementing the programmable logic component and, on the other hand, an executable file for orchestrating the operation of the programmable logic component;

[0061] A step 13 of creating the programmable logic component based on the binary file (for example, operation of flashing the “bitstream” binary file onto an FPGA, or even creating an ASIC based on the binary file); and

[0062] A step 14 of loading the executable file by the general processor, for the implementation on this processor of orchestration software allowing on-the-fly configuration, without modifying the “bitstream”, of the programmable logic component within genericity limits set by the implementation binary file.

[0063] These steps can lead to the manufacturing (initialization) of an integrated execution platform (of the SOC type, from the English for "System On Chip") comprising both the general-purpose processor, level 1 memory accessible by the general-purpose processor, the programmable logic component from the parameterization file (this component having resources determined from the parameterization file and having a level 0 memory). As described and illustrated later in relation to figure 5a, other types of architectures can however be implemented for the implementation of such a platform, including in particular a type of architecture based solely on programmable logic (i.e. not requiring the presence of a general-purpose processor).

[0064] Equipped with this programmable logic component and more generally with this implementation platform, the invention then relates to the manner in which this component is used within the platform, in particular to configure and execute the various artificial intelligence functionalities which are implemented therein. The invention relates in particular to the manner in which the programmable logic component receives the configuration data and executes the artificial intelligence algorithms according to the directives provided by the software running on the general-purpose processor, by performing level 0 and level 1 memory writes and reads.

[0065] The following sections detail the various characteristics of the pseudogeneric component and the associated platform.

[0066] 2. Generalization of calculations

[0067] First, the inventors have defined a formalization of operations carried out within the framework of a series of calculations. The calculations in question may be matrix calculations, but not only. More particularly, it is possible to describe the methodology implemented by the inventors based on the particular example of neural networks. Thus, on the basis of the findings of inefficiency, particularly energy inefficiency, detailed previously, the inventors have defined a generic description of neural networks (a) having online training capabilities, (b) allowing on-the-fly configuration and reconfiguration for any type of neural network, (c) portable on any embedded device but particularly targeting FPGA / ASIC devices and (d) allowing the possibility of integration into a federated or distributed learning architecture.

[0068] The basis of the inventors' approach is the assertion that all neural networks are, in fact, transformations of one multidimensional space (the input dimensions) into another (the output dimensions). Generally speaking, any computation is a transformation of a multidimensional input space into a multidimensional output space. Optionally, these input and output spaces can be one-dimensional (a polynomial transformation in a single variable, for example).

[0069] More specifically, two main global operations are used for any neural network: inference: this is the prediction operation, it is generally the part that is executed on the embedded electronic component once the neural network has been parameterized and trained; this inference can be broadly defined as the resolution of an equation from a plurality of input variables; learning: this is the learning operation composed of two phases. Gradient descent in which the goal is to propagate the error between the output (generated by the neural network) and the expected output (label) to all layers and the second phase where the objective is to correct the transformation in order to make it more optimal for the next inference step (to simplify, updates of the transformation coefficients).

[0070] These two phases are described below, and we see that they are both equivalent to a multidimensional matrix calculation, comprising a set of input and output parameters (input dimension, output dimensions) and internal parameters (dimensionality of the hidden layer matrices, weights - i.e. values ​​- of these matrices, sizes and values ​​of the biases and selected activation functions).

[0071] 2.1 Inference operations (feedforward pass)

[0072] For the full inference phase (i.e., the feedforward pass) of a complex neural network, considering that Output = Y, Input = X, and W the internal data (internal to the full transformation, usually called the weights), it is possible to write that a neural network is a function f(..) from X and W to Y:

[0073] Output = NeuralNetwork Input, InternalData) <=> Y = f(X, W)

[0074] Or more formally,

[0075] X represents a tuple of real numbers in the set R n (in n dimensions), W represents an n-tuple of real numbers in the set R k (k-dimensional) Y a tuple of real numbers in the set R m (with m dimensions). The complete inference transformation can be decomposed as a combination of N unitary transformations and it is possible to write:

[0076] The results provided by these functions f N , are stored in memory as a hidden layer denoted H. Therefore:

[0077] So, for example, an MLP layer (extended and complex version of the original perceptron) is a transformation from a 1-dimensional vector to another 1-dimensional one via matrix multiplication, vector addition and the use of a non-linear activation function. Considering that this transformation is denoted MLP and is applied from a vector X of N values ​​to a vector Y of M values. The MLP layer consists of multiplying X by a weight matrix W, adding a bias vector b of M values ​​(note that W and b are considered as the internal data globally denoted W for the transformation) and passing all the results into an activation function denoted <f>.

[0078] Y = MLP(X, VT) Y = <P(W X X + h) Par exemple, si l'on combine deux fonctions MLP, il est possible d'écrire que :

[0079] Describing neural networks in this way, we can build any type of neural network. For example, the Lenet5 architecture, which, using: C for the convolution layer transformation, SS for the downsampling transformation and FC for "fully connected" (i.e. MLP), X for example an MNIST image as input and Y for the prediction of handwritten digits, can be written:

[0080] Y = MLP7MLP6MLP5SS4C3SS2C1X; W1')')')')')')')')

[0081] The transformations (i.e. all the different functions f) can be any type of neural network cell, for example:

[0082] MLP a 1-dimensional to 1-dimensional transformation.

[0083] RNN cells such as LSTM, GRU, MGU, STAR or SRU which are a transformation from 1 dimension to 2 dimensions (as an illustration, we can see an MLP with a time dimension).

[0084] CNN (multidimensional to multidimensional) cells or combined CNN-RNN such as ConvLSTM.

[0085] Simple operations: Buffer (example 1D to 2D such as from an MLP layer to an RNN), Addition or aggregation layer (1D to ID, 2D to 2D, ...), Downsampling (Average, Max, ...)

[0086] Others: Extended Kalman Filter (EKF) or Uncentered Kalman Filter (UKF) Self-organizing maps (SOM), Markov decision processes (MDP), POMDP, ...

[0087] 2.2 Backpropagation Operations

[0088] On the other hand, to enable the neural network to learn, we need the inverse operations to update all the functions / according to the error between the generated output Y (from the neural network inference pass) and the expected output Y*, error noted 6Y:

[0089] SY = Error(Y *, Y)

[0090] The goal of backpropagation is to update the internal data W for each function / in order to obtain a better performing neural network in the next iteration (i.e. with a lower error). The very first step is to calculate the gradient of all hidden layers, the inverse of inference from output to input, for example: SX = f'(SY, W')

[0091] SX = SH0= f (SH^ W SH = f'2(SH2, W2)

[0092] SH N-1 = f' N (ÔY, W N )

[0093] These gradients are calculated from the description using a propagation rule (Daisy Chain Rule), then used to update the internal transformation data using an optimization function, denoted optim, such as Stochastic Gradient Descent (SGD) normal and with momentum, ADAM or ADAMAX. These algorithms are based on a set of parameters denoted P (e.g. the learning rate, these parameters can be different for each layer).

[0094] SW = optim(W, SH, P)

[0095] SWN = optim(W N , SY, P N )

[0096] So, in this gradient descent calculation, the objective is to update the internal data W, which correspond to the weights of the matrices as well as the biases. The learning (in which the backpropagation operation is carried out) is therefore again a matrix calculation.

[0097] Consequently, for the example of neural networks, their implementation consists of setting a certain number of parameters (number of matrices, their dimensions, weights, biases, activation functions, optimization functions, etc.) which make it possible to completely implement a network in the form of a particular component. On this basis, the inventors have worked to formalize a certain genericity of these parameters in order to implement, within a component, a set of generic functions allowing the implementation of a much larger number of neural networks, and more generally a much larger number of mathematical functions.

[0098] 2.3 Memory Management

[0099] However, the conceptual view of multiple dimensions is not applicable to the memory layout of an architecture based on an FPGA, an ASIC, a CPU or a GPU. All dimensions must be converted into a single dimension. Thus, a 3-dimensional entity (matrix) (DIMI, DIM2 and DIM3) is stored in a simple format (all elements are stored in rows, and addressable in rows) with for example DIM1 first, then DIM2 and finally DIM3. This layout could have been implemented in another order DIM3 first, then DIM2 and finally DIM1. The choice of this layout of the data to be stored in memory is important to consider in order to access it in the most efficient way possible, especially in the case of an architecture with multiple memory levels.Thus, on a (physical) device, according to the technique developed by the inventors, this memory is used to read the available data (inputs X and weights W) according to their location in the memory for a transformation f and store (i.e. write) the output Y to a specific memory location. The logic of the transformation (all necessary operators) is implemented on the pseudo-generic component. The latter accesses the memory according to access parameters provided to it via configuration data transmitted by means of the orchestration software, loads the data into memory, performs the requested operations on this data according to the provided execution parameters, and writes the obtained results to a specific address in its internal memory (the execution parameters and the specific address also being part of the configuration data transmitted by means of the orchestration software).

[0100] In other words, for a given pseudo-generic component, the implementation of a matrix operation consists of loading, row by row or column by column, the matrices involved in the matrix operation (for example a multiplication), then performing the operation on each term of the matrix and finally recording each of these terms in a resulting memory area of ​​the pseudo-generic component. Other mathematical operations can also be applied to the terms of the matrix before their recording if necessary.Thus, according to the invention, the implementation (use) of a pseudo-generic component to manage several different mathematical operations consists of the execution of a method comprising as illustrated in relation to Figure 2, in a particular embodiment: a step 21 of configuring the component using configuration parameters; the configuration parameters define the way in which the component must execute; for a neural network, for example, the configuration parameters define the sizes of the different layers, the activation functions to be used, the order of the layers, etc. and at least one iteration of the following steps: a step 22 of loading a set of execution parameters: these are typically the parameters of the different functions, such as for example the numerical constants, factors, etc.; in the case of a neural network, this typically involves loading the weights of a matrix, the activation function, and any associated biases for the layer being executed; a step 23 of loading a set of execution variables; this typically involves loading the input values; in the case of the neural network, this is the input vector or matrix of the layer being executed; at least one step 24 of calculating at least one execution result as a function of the set of execution parameters and execution variables; for a conventional mathematical function, this involves calculating the image of the set of execution variables using the execution parameters; for the case of a neural network, this involves calculating the result matrix of the current layer; a step 25 of recording, within a memory area of ​​the component, said at least one execution result.

[0101] The different steps of this process are implemented as many times as necessary to complete all the calculations. In the case of the neural network, these steps are implemented until, for example, one or more output vectors are obtained depending on the input variables.

[0102] 2.4 General Implementation

[0103] Building on these developments, the inventors have therefore developed a method for creating a pseudo-generic component, within a platform, allowing the implementation of several types of calculations by this single pseudo-generic component. It is then possible to carry out several different types of calculations (for example, implement several different neural networks) without it being necessary to modify the pseudo-generic component itself. This creation method is based on the implementation (manual or automated) of a configuration file which makes it possible to frame (define) the possible functionalities of the component.This configuration file is used to generate, from a compiler, two files with related purposes (which explains their generation from the same configuration file): a binary implementation file of the "bitstream" type, used to define the programmable logic of the pseudo-generic component (FPGA or ASIC) and more particularly to define the types of calculations executable by the programmable logic component (in other words the genericity limits of this component), and a file executable by a processor allowing subsequent configuration, on the fly, of the programmable logic component, within the limits of the types of calculations executable previously fixed by means of the binary implementation file.The final implementation architecture of these two files is a platform comprising a general-purpose processor (CPU), a memory managed by this processor and a generic component, the general-purpose processor and the generic component exchanging data via at least one data bus.

[0104] The file executable by the processor is a so-called orchestration file: it allows the implementation on the processor of orchestration software to manage the configuration of the programmable logic component, the data necessary for the implementation of this component (for example the training data for a neural network, which are the weights of the matrices, the variables, etc.) from the memory available for this processor. More particularly, in this architecture, the component (initially defined by its "bitstream" file), receives, from the processor and more particularly from the orchestration software, the necessary configuration data (for example in the form of a configuration file), then once configured, accesses in L1 memory the execution data (input variables, weights of the layer(s) of the network, etc.).The component executes the operations instructed to it by the processor (matrix operations). More specifically, the configuration data (i.e. the configuration file) defines memory locations (and their structures) in which to read and write for the implementation of a particular calculation typology depending on a targeted application, among the set of executable calculation typologies available because previously fixed by means of the implementation binary file.

[0105] In other words, the present technique is based on what can be considered as a two-stage configuration of the programmable logic component: a first configuration phase, based on a binary implementation file (or bitstream) which aims to set limits on the types of calculations that can be executed by the programmable logic component (i.e. genericity limits); a second configuration phase, which can be repeated as needed, which consists of using orchestration software to transmit configuration data to the programmable logic component allowing it to be configured more precisely for a given application, within the limits of the types of calculations that can be executed previously set during the first configuration phase.

[0106] We thus have a "configurable" platform comprising the general processor, the memory associated with this processor (called level 1 memory - L1), a "generic" component itself having its own working memory (called level 0 memory - LO). Such a platform can for example be implemented within an "intelligent" vehicle. Thanks to the "generic" component, it is possible to configure new functions to be fulfilled, for example 1 of the new artificial intelligence algorithms (which are matrix calculation algorithms as explained previously), which were not originally planned during vehicle manufacture.

[0107] In the following sections, an example of the implementation of such a platform oriented towards the processing of artificial neural networks (i.e. matrix calculation) is described. The inventors have developed the described method to have, within a "pseudogeneric" component, the functionalities necessary for the implementation of several typologies of neural networks while limiting the impact of this genericity.

[0108] 3. Initial parameterization of the pseudo-generic component and the associated platform

[0109] Creating a combined platform includes a first parameterization step discussed in this section. The very first and initial step of this parameterization defines the "genericity limits" of the component. Since genericity comes at a cost, designers must define what the pseudo-generic component is capable of doing. An example of a parameterization file is illustrated in relation to Figure 3, in a particular embodiment.

[0110] This file is completed by a user to generate a pseudo-generic component and its associated software. The user determines within this configuration file which set of operators, from a predetermined set of available operators, should be included in the component. He also sets maximum values ​​for certain parameters.

[0111] The parameters in this parameterization file are used to generate synthesizable VHDL code from a completely generic description of a component. By determining the operators and parameters, the user can adjust the resulting pseudo-generic component to their specific needs and remove unnecessary logic. The logical size (in an FPGA or ASIC) is then reduced. The parameterization part, as illustrated in Figure 4, allows two entities to be defined:

[0112] (a). The COBANN SW entity which is the orchestrator;

[0113] (b). The COBANN HW entity that contains the main invention. It is the artificial neural network coprocessor that performs the calculations of the expected neural network based on the method described previously.

[0114] The inventors thus obtain a dual architecture comprising a “Processing System” (PS), i.e. the processor, associated with the logic (PL running on an FPGA or an ASIC) as described in figure 5.

[0115] PS: Processor-based processing system (CPU);

[0116] MGR: Manager (processing software) for configuring the programmable logic component; DH: Data manager;

[0117] CH: Configuration Manager;

[0118] TH: Training Manager;

[0119] PER: Peripherals;

[0120] PL: programmable logic component; FPGA or ASIC component;

[0121] El: executor 1 within the programmable logic component;

[0122] E2: executor 2 within the programmable logic component;

[0123] FS: file system access;

[0124] MC: Memory Controller;

[0125] The component described in Figure 5 includes two entities. A Manager entity (also called "software component") which is in charge of the overall management (configuration, data provision) as well as the connections with the different PER peripheral interfaces (for example cameras or sensors on a real embedded system). Another entity is the Executor component El, E2 (also called "hardware component") which can be unitary or multiple (two are shown in Figure 5). It is the Executor which is configured by the Manager thanks to the configuration described previously. These components are illustrated in Figure 5 which describes a Manager connected to two Executors. One of these executors has the ability to do online training (on the component) so it is capable of doing back-propagation (i.e. BP capable). The other is only capable of doing inference (i.e. feedforwardFF only) and therefore assumes offline training (according to the offline training paradigm). As a reminder, the training capabilities of a component are defined during the parameterization step, described previously.

[0126] Thus, once configured and synthesized, the pseudo-generic hardware component is then implemented in an FPGA, or in an ASIC and is controlled by the PS processor (the whole can be integrated within a SOC). This hardware component is considered to have at least two memory levels, namely LO and Ll. As previously explained, the LO memory is the internal memory of the hardware component such as block memories (BlockRAM - BRAM) on FPGA. The Ll memory is the external memory accessible via the PS processor such as DDR memory. Thanks to this extended memory, and the management (configuration) of this memory by the software component (running on the PS processor), the inventors allow the use of larger neural networks. It is important to note that, due to its hardware construction, all PS and PL entities access the same DDR memory denoted Ll.This implies that the data accessible to the Manager on the PS is also accessible to the Executors of the PL. This is important later in explaining when considering the interactions between the Manager (on the PS) and Y Executor (on the PL).

[0127] It is also possible to implement this architecture on FPGAs (and ASICs) only. In this type of architecture, the Manager component is directly integrated into the logic of the FPGA (or ASIC) as well as the memory controller. This component is then not necessarily interfaced with peripherals, nor with a file system. All transactions for configuration and data management can be carried out via an Ethernet interface (in 1 Gbits for example), thus opening the possibilities towards distributed neural networks.In other words, although mainly described in relation to an electronic platform (or pseudo-generic component) comprising a general-purpose processor and a programmable logic component in the present document, the present technique is not limited to this architecture and can also be implemented with other forms of architecture, provided that these allow the implementation (possibly on the same physical component) of at least one Manager entity and one Executor entity as previously introduced.Figure 5a illustrates another example of architecture, in which a Manager entity (MGR) and two Executor entities (El and E2) are implemented on a single programmable logic component (PL), and in which the Manager entity interfaces with the peripheral (PER) (via Ethernet for example) to receive the various data (configuration data for example) in order to store them in the L1 memory via a memory controller (MC).

[0128] The pseudo-generic component includes several interfaces, as explained in Figure 6 and Figure 7. An Executor entity has several interfaces in order to function. A component with training capabilities has more interfaces than the component that is only capable of inference. A capable BP executor (Figure 6) has thirteen different interfaces. There are nine used for inference (Feed-Forward i.e. FF, Figure 7): (1) configjn, (2) datajn, (3) result_out, (4) transf_data_inout, (6) layers_data_inout, (7) opcodejn, (8) localstate_out and (9) memory_layout_out. There are 4 used for learning (Back-Propagation i.e. BP): (10) labeljn, (11) dtransf_data_inout, (12) dlayers_data_inout and (13) random_table_in. Several interfaces are only used for training neural networks (when these features are implemented).Therefore, they are not present when the component is configured only for inference (calculating predictions).

[0129] [configjn] contains the entire configuration of the deep learning entity (neural network). It specifies the list of transformations to be applied from input to output. It defines the characteristics of each of the operators (i.e., transformations) such as the activation function and how and where to read the different active operands for this transformation.

[0130] (i.e. transformation data and layer data).

[0131] [datajn] is the data that is presented to the pseudo-generic HW component in order to be processed. It can be a single sample (i.e. an image, a stream, etc.) or multiple sample inputs if the network allows it to backpropagate for training (see parameterization section) (denoted by X in the generalization of calculations).

[0132] [result_out] This is the output of the feedforward calculation of the neural network (denoted Y in the generalization of calculations).

[0133] [transf_data_inout] This contains all the transformation data (weights) of the ANN

[0134] (noted W in the generalization of calculations).

[0135] [layers_data_inout] These are the intermediate results of each layer (denoted H in the generalization of the calculations).

[0136] [report_out] Various indicators used to determine network performance.

[0137] [opcodejn] sent by the PS to execute the pseudo-generic HW component. This can be (non-exhaustive): configuration, inference, training.

[0138] [localstate_out] Returns the state of the pseudo-generic component. It can be (non-exhaustive): parameterized, Target error reached, Target error not reached.

[0139] [memory_layout_out] the format and dimensions of the outputs. Since the memory layout of each layer can be deduced from the transformations, the programmable logic component provides the layout of the results to the PS processor.

[0140] The training interfaces for neural networks are:

[0141] [labeljn] This is the label of the training data, i.e., it is the expected output for the neural network. It must be compared with the result above to generate an internal optimization error of the network via backpropagation (denoted by Y in the generalization of calculations).

[0142] [dtransf_data_inout] These are the delta coefficients, used for calculating the gradient (noted W in the generalization of the calculations).

[0143] [dlayers_data_inout] These are the delta layers, used for calculating the gradient (denoted by H in the generalization of the calculations) [random_table_in] a table of random values ​​(generated by the software running on the PS processor), to mix the training samples.

[0144] Thus, the initial parameterization makes it possible to have a pseudo-generic component associated with a general-purpose processor.

[0145] The Executor interfaces described are, in operational conditions, buses for accessing memory areas of the external L1 memory (i.e. the DDR). It should be remembered that, on SoC implementations, this data is accessible via the PS memory manager (the CPU). As illustrated in Figure 8, the role of the Manager (MGR) is to provide the Executor (E2, for example) with the memory base addresses (Adr_Bse) on which it can read / write the data linked to these different interfaces. The structure of the data from these base addresses is determined from the configuration as explained later.

[0146] 4. Configuration of the pseudo-generic component

[0147] As described in the computational generalization section, a neural network is considered as a combination (usually a succession) of transformations applied to an input in order to generate an output. The configuration of the programmable logic component allows defining this succession of transformations from input to output. The memory layout of the input must also be defined. For each transformation, the user can define the type of transformation (see a list below), the type of the activation function, the data layout of the input (of the layer). The different parameters of the backpropagation layers can also be defined, if they were included during the parameterization phase.The configuration file includes for example an identifier, a neural network size, a structuring of the input data (size and dimension for example), the structuring of the layers (for each layer) [type, format of the data - internal, input, output -, activation function to use, dimension of the vectors or matrices], the structuring of the back-propagation, for each back-propagation layer (type of error, parameters, learning rate, moment, regularization), the number of cycles ("epoch"), etc.

[0148] In summary, the configuration is used to define the types of operators (i.e. transformations) to be applied and its characteristics, and to which operands they apply (internal transformation operands like weights and data from the previous layer). The configuration can be stored in a file. The PS processor performs an initialization of the pseudo-generic component with this configuration before the calculations. This configuration is defined by the user, it can be determined using popular artificial intelligence tools (frameworks) such as Pytorch or TensorFlow. Non-exhaustive list of inference parameters (feedforward):

[0149] Transformation type: conv, sub max, sub min, sub avg, sub max, conv w sub, FC, LSTM,

[0150] STAR, GRU, MGU, memory transformation, aggregation

[0151] Activation function: none, sigmoid, tanh, tanh opt, sig int, tanh int, relu, softmax

[0152] Non-exhaustive list of backpropagation parameters:

[0153] Descending gradient type: Stochastic Gradient Descent (SGD), Stochastic Gradient Descent with Momentum (SGDM), adam, adamax, etc.

[0154] Usually, the internal memory of an FPGA device (or even an ASIC) is limited and / or extremely expensive. As mentioned above, the approach adopted is not limited to local memory since it is possible to use external memory components that allow via write / read operations to load and export the data necessary for each operation. Burst access from LO to L1 memory is defined in the parameterization phase.

[0155] 5. Memory exchange processing / Component configuration

[0156] 5.1. Generating a configuration and data in L1 memory

[0157] Component configuration, once the component has been "instantiated" (i.e. once the genericity limits of this component have been defined and a functional component is available), consists of allowing the component to interact correctly with the L1 memory on each of its memory bus interfaces (as defined previously). A configuration, within the genericity limits of the parameterization, is composed of three main elements: inputs, labels and transformation lists. If we only consider neural networks doing inference, then the configuration does not include labels. We are interested here in the data presented on the configjn interface. This configuration is for example generated from an XML file whose formalism, structure and elements have been defined.This XML file is given as input to the Manager component which generates the correct data structure(s) for the configjn interface. It is quite possible to take into account ONNX (Open Neural Network Exchange) type configurations or even specific files coming from the most used frameworks (development frameworks) such as Pytorch (and its models saved in "*.pth"). If we consider offline training components, the model descriptions integrate the data of the different transformations specific to the neural network. Thus, the transf_datajnout interface is also constituted in accordance with the configuration of the configjn interface.In other words, the manager, based on a configuration file (configjn interface), informs the hardware component (Executor) of the type as well as the structure and location (memory addresses) of the transformation data (the data of the different layers present on the transf_data_inout interface) in the L1 memory. This configuration also allows the executor to know the structure and location of the input data (inputsjn and possibly labelsjn if the component is capable of training). Thus, the executor can implement the functions planned for the neural network from the planned configuration while correctly accessing the necessary data from the data structure and base addresses also provided by this configuration.Then, the executor can perform the neural network calculations by writing / reading the intermediate data for the different layers in the L1 memory (via the layers_data_inout interface) and provide the final result on the result_out interface. The executor can also provide the data structure (addresses and dimensions) for the data it generates via the memory_layout_out interface.

[0158] 5.2 Configuring the Executor

[0159] A configuration is thus provided to the Executor component from a given base address. The Manager provides it, via the configuration file transmitted via the orchestration software, the different addresses to which these different interfaces will be connected and used. The Manager also transmits to it by means of this file an opcode (operation code) to indicate to it that it must configure itself from this data present on its configjn interface at the base address provided. The Executor therefore determines from the configuration provided (containing the description of the inputs and transformations, as well as the labels) the way to correctly read and write the data and the results in the DDR (the L1 memory).The component also provides on the memory_layout_out interface the structure of the data it generates (typically the outputs and data for the intermediate layers, so that the MGR manager can read and decode them). At the end of this configuration phase, the Executor returns its configuration state and if this state is correct, then the component is ready to run and is configured from the provided description.

[0160] In other words, the Executor reads the configuration from the addresses provided by the Manager, then extracts, from the provided addresses, in L1 memory, the data structures to be used and the appropriate functional characteristics, then determines: how it performs the readings and writings in L1 memory and what is the functional logic to be applied (MLP, CNN, activation functions, etc.).

[0161] 5.3 Examples

[0162] The implementation of the method of the invention is illustrated using a concrete example. To do this, we assume that we have three different neural networks that we wish to apply from the same input data (here, the historical case study of MNIST). These networks are (1) ANN-1 is a multi-layer MLP network; (2) ANN-2 is a Lenet-5 type CNN network and (3) ANN-3 is an LSTM type RNN (recurrent) network. The following hypotheses are formulated:

[0163] These three networks are capable of solving the MNIST classification problem. They take as input a gray image of size 28x28 and provide as output a classification vector of size 10 which corresponds to the probability of the recognized digit on the image provided as input (between 0 and 9);

[0164] The considered DDR (thus the L1 memory) is large enough to contain all the data of the three networks. This means that the configuration data, the data for the intermediate levels and the transformation data for each network are available in L1 memory (this assumption is purely rhetorical since many DDRs for embedded cards can reach 16, 32 or even 64 Gbits, which is quite sufficient to deal with this illustrative problem).

[0165] The three networks were trained offline using a tool (Pytorch, Tensorflow, ...) and were saved in ONNX format in the files annl.onnx, ann2.onnx and ann3.onnx. These files were provided to the Manager entity in order to place the data and associated configurations in the Ll memory. The Executor was correctly configured with the logical functionalities required for the execution of these three types of networks.

[0166] The Executor has limited logic, so it can only handle the execution of one neural network at a time (one can configure and execute ANN-1 or ANN-2 or ANN-3).

[0167] Thus, after a step of generating the configuration in memory L1 and copying the associated data, as illustrated in relation to figure 9 (for the ANN-1 network), figure 10 (for the ANN-2 network) and figure 11 (for the ANN-3 network) the Manager can configure the Executor so that it executes each of the networks available in memory L1.

[0168] The Executor provides, successively, for each of the three networks, on the interfaces "config_in", "transf_data_inout" and "layers_data_inout", "data-in" the data necessary for the implementation of the network. Once configured with this data, the Executor performs the calculations for which it was configured, including matrix calculations and the activation functions of the implemented logic. The data is read / written in L0 memory to perform the calculations "internal" to the Executor and when these calculations are completed (or when the memory must be purged (for example for a possible following calculation cycle), the data is written in L1 memory (on the interfaces "result_out" or "memory_layout_out"). The Executor reads / writes its operating data at different addresses in L1, the associated configuration allows it to interpret the memory structure linked to these inputs / outputs.It can be noted that in this specific case, the component reads the input data (datajn), writes its output data (result_out) and provides the structure of its memory interfaces (i.e. memory layout on the memory_layout_out interface) from the same addresses (therefore the same memory areas). It may be relevant to use different areas in particular if the neural networks used (and to be configured on the component) have different objectives / goals.

[0169] 5.4 Execution of L0 / L1 interaction management

[0170] Once configured as described above, the Executor must implement the operations for which it has been configured and for this it performs several "copy" operations in order to acquire all the data (operands) necessary to carry out the operations configured in Ll in order to have them available in LO memory. Conversely, once the operations have been carried out, it performs copy operations of the results obtained (by its calculations) from its local LO memory to external Ll memory. It is possible to illustrate this operation by considering the elementary operation of an MLP network in the form of expression 1 considering the matrices or in the form of expression 2 considering the elements (the bias b is a particular weight forming part of the transformation data).

[0171] The vector X is a vector of size N accessible on the interface "layers_data_inout" of the memory L1 and likewise the vector resulting from this operation noted Y (of size M). The transformation data W and b are accessible on the interface "transf_data_inout" of the memory L1. If we consider that N = 1000 and M = 500, we can determine that the weight matrix W is of size M * N = 500 * 1000 and the bias vector is of size M = 500. A possible division is therefore to copy from the memory L1 to the local memory L0 the data necessary to calculate the elements of Y in a unitary way. Thus step by step, each element y7 is calculated by following the following method, as illustrated in figure 12 and figure 13 which respectively present the calculation of the element Yi then of the element Y2:

[0172] (1) Copy data from Ll to L0,

[0173] (2) Calculation and generation of the result from the implemented logic; and

[0174] (3) Copy the result from memory L0 to external memory L1.

[0175] Note that the copy of the input vector X is done only once for all calculations of Y (because this input vector X does not change).< / f>

Claims

CLAIMS 1. Electronic data processing platform, platform comprising a general-purpose processor (PS) and a so-called level 1 memory (Ll), said platform further comprising a programmable logic component (PL) comprising a so-called level 0 memory (LO), the programmable logic component being parameterized, by means of a binary file for implementing said programmable logic component, so as to define typologies of executable calculations, said platform further comprising orchestration software implemented by means of an executable file running on the general-purpose processor, the orchestration software being responsible for providing, to the programmable logic component, a set of configuration data for its execution within the limits of the typologies of executable calculations defined by said binary file,said configuration data comprising said binary file and said executable file being generated by compiling a configuration file defining a set of operators included in said programmable logic component from among a predetermined set of available operators., 2. Electronic platform according to claim 1, characterized in that the programmable logic component is an FPGA or an ASIC.

3. Electronic platform according to claim 1, characterized in that the programmable logic component is configured so as to perform inference calculations for artificial intelligence.

4. Electronic platform according to claim 1, characterized in that the programmable logic component is configured so as to execute learning calculations for artificial intelligence.

5. Electronic platform according to claim 1, characterized in that the orchestration software is arranged to provide the execution component with a level 1 memory address at which the configuration data set is accessible, and to provide the programmable logic component with an execution command. Tl 6. Electronic platform according to claim 5, characterized in that the programmable logic component is further configured to extract the data structures and the functional characteristics of the calculations to be executed and configured to determine how to read and write data in level 1 memory (L1) and determine an execution logic to be implemented.

7. Electronic platform according to claim 6, characterized in that the execution logic to be implemented belongs to the group comprising: a multi-layer perceptron type neural network; a convolutional type neural network; a recurrent neural network.

8. Electronic platform according to claim 5, characterized in that the programmable logic component is configured to copy, within its level 0 memory (LO), said configuration data set and to perform the calculations required by the configuration data of the configuration data set.

9. Electronic platform according to claim 6, characterized in that the programmable logic component is configured to recursively load, into the level 0 memory (LO), input data from the level 1 memory (L1), then to calculate, from said input data, output data and to copy said output data into the level 1 memory (L1).

10. Method for obtaining an electronic platform according to claim 1, characterized in that the creation of the programmable logic component comprises the following steps: obtaining (11) a parameterization file of the programmable logic component, said parameterization file defining a set of operators to be included in said programmable logic component from a predetermined set of available operators, said parameterization file comprising, for at least one characteristic of said programmable logic component, a value representative of the capacity of the programmable logic component to have this characteristic; compiling (12) the parameterization file of the programmable logic component, delivering on the one hand a binary implementation file of the programmable logic component, and on the other part an executable file for orchestrating the operation of the programmable logic component; creation (13) of the programmable logic component based on said binary file; loading (14) of said executable file by the general processor, delivering software for orchestrating the operation of the programmable logic component.

11. Method for using the programmable logic component of an electronic platform according to claim 1, characterized in that it comprises a step of configuring (21) the programmable logic component using configuration parameters, and at least one iteration of the following steps: loading (22) a set of execution parameters; loading (23) a set of execution variables; calculating (24) at least one execution result as a function of the set of execution parameters and execution variables; - recording (25), within a memory area of ​​the programmable logic component, said at least one execution result.

Images