SOC programmable logic device

Abstract

A System-on-Chip Programmable Logic device, SoC PL (2), for performing a distributed and diverse inference task, the SoC PL (2) comprising a Host Processing System, PS (3), configured to receive an external request which relates to the inference task, and is accompanied by input data, and load configuration data of a deep neural network, DNN, corresponding to the received inference task upon receiving the external request; and a Programmable Logic, PL (7), including an input buffer (9) configured to store the loaded configuration data from the PS (3) under control of the PS (3), and a plurality of processing elements, PEs (11), configured to perform inference on the input data under use of the DNN when the PS (3) indicates that the input buffer (9) is ready, wherein the PEs (11) form a systolic array.

Description

[0001] DESCRIPTION

[0002] SOC PROGRAMMABLE LOGIC DEVICE

[0003] TECHNICAL FIELD

[0004] The invention relates to concepts for edge artificial intelligence (Al) devices and applications thereof and in particular to a System-on-Chip Programmable Logic device (SoC PL) and a computer-implemented method of the SoC PL for performing a distributed and diverse inference task, a computer-implemented method for synthesizing and running an SoC PL, a computer program related to the methods and a data carrier storing the same.

[0005] The following background is intended solely to provide information necessary to understand the context of the inventive ideas and concepts disclosed herein. Thus, this background section may contain patentable subject-matter and should not be regarded as prior art per se.

[0006] BACKGROUND

[0007] Internet of Things (loT) offers significant benefits to business operations through the instrumentation of physical operations with high fidelity sensors, the tracking of events in operations with high frequency, and the conversion of sensor data into actionable analytic insights through software and services. However, the physical settings of day-to-day operations (such as factories, warehouses, retail stores, and other facilities) are highly heterogeneous in terms of sensor hardware, networking topology, data protocols, and data formats. This heterogeneity presents significant challenges for software development, testing, and deployment. Code modules must be redeveloped for each combination of hardware, networking, data protocol, and data format, resulting in high costs.

[0008] Local operations in distributed network loT environments may encompass a multitude of different types of industrial machines and processes in a plethora of different business and technology sectors. These sectors include, but are not limited to, manufacturing, oil and gas, mining, transportation, power and water, renewable energy, healthcare, retail, smart buildings, smart cities, connected vehicles, and even consumer wearables. In an loT environment, and particularly in an Industrial Internet of Things (IIoT) environment, numerous sensors and other data-generating sources may be deployed in connection with an enterprise's local physical operations to detect and track various physical conditions, occurrences, and events continuously or at very high frequency, or any combination of these. Such sensors and data-generating sources can produce vast quantities of raw data that require processing and analysis to be of any use. Moreover, as the costs of sensors and network technology decline, the number and distribution of sensors in local networked operations is growing rapidly.

[0009] In an loT environment, enterprises may rely on remote data center or “cloud” infrastructure to host the computing infrastructure and software applications needed to process and analyze the locally generated sensor and device data in order to exploit economies of scale and system efficiencies. However, remote cloud-based computing and storage solutions have a number of shortcomings. Firstly, remote data centers can be located at a considerable distance from the locations where an enterprise conducts its business operations, such as factories, warehouses, retail stores, and others. This can result in the generation of large volumes of sensor and device data at these locations. It is not always feasible to transfer such large volumes of data to a remote cloud computing or storage site, or a combination of both. Furthermore, connectivity may not always be reliable or even available. Even in the event that connectivity is available, the available bandwidth is likely to be insufficient for the volumes of data generated. Furthermore, even if sufficient bandwidth is available, it is likely to be cost prohibitive in consideration of the volumes of data to be transferred.

[0010] Even if connectivity, bandwidth, and cost were not issues, remote cloud-based computing and storage sites lack the capacity to process and analyze the data generated in local environments in real-time, make decisions based on that data, and provide potentially critical guidance or warnings back to the local environments in a timely manner. To illustrate, a cloud-based computing site may receive, process, and analyze data from a sensor monitoring a machine in an enterprise's local facility. However, by the time the site determines that the machine's operating characteristics have reached a limit requiring immediate intervention or predicts a need for maintenance, the machine may have already suffered significant damage.

[0011] The high volume, diversity, and complexity of streaming sensor and device data generated locally present challenges for interpreting the data and obtaining predictive and other insights therefrom. Cloud-based data mining and machine-learning model generation, training, and application can be employed to address these challenges. However, cloud-based machine learning lacks real-time responsiveness that may be critical in some situations in local environments. Furthermore, the absence of contextual information, such as local domain knowledge or expertise, renders it challenging to derive actionable insights from data, even when the entire data set is transmitted to and analyzed in the cloud. However, the implementation of machine learning at the edge, where locally generated data can be received and acted upon directly, in real time, with local context, and without awaiting transmission to a remote cloud site, would allow actionable insights to be derived from the locally generated data and made available for use locally in real time.

[0012] Typical machine learning models are designed to operate in a cloud environment, which presents a number of challenges when attempting to implement and adopt machine learning at the edge. Typical cloud-based machine learning models necessitate a substantial computing capacity, whereas edge-based computing devices typically possess a limited computing capacity. Typical cloud-based machine learning models are also capable of operating on batched data, such as that typically stored in cloud-based storage. However, they are not able to function in real-time on high velocity, high volume streaming data, such as that typically produced by sensors in industrial and other loT environments. Additionally, typical cloud-based machine learning models function optimally when data has undergone pre-processing to eliminate errors and gaps, normalization, and other similar operations. However, they may not provide accurate results when processing raw sensor data. Moreover, since typical cloud-based machine learning models operate on batched data, they may not be sufficiently efficient or rapid enough to keep pace with the high rates and volumes of streaming data that are typically generated in industrial and other loT environments. Moreover, while typical cloud-based machine learning models can be readily updated or tuned in the cloud as additional batches of data are received and analyzed, a distinct approach is required for updating remote edge-based machine learning models.

[0013] Consequently, there is a need for the development of enhanced computing systems, architectures, and techniques within the context of the loT environment. In particular, there is a need for an improved intelligent edge computing device with machine learning capability for use in various loT environments. This device must be able to efficiently and effectively receive, process, and analyze high-rate, high-volume streaming data. It must also be able to derive analytical meaning and predictive insights from this data and take appropriate local actions in real-time, without the need to transfer data to a remote cloud or other facility for processing and analysis. Additionally, there is a need for an edge device on which machine learning applications can execute efficiently and effectively, despite limited compute and storage resources. Additionally, there is a need for an edge device that can facilitate the development, training, evaluation, deployment, and updating of machine learning models for the edge in an efficient manner, in conjunction with remote cloud resources.

[0014] Thus, the invention concerns one or more of the above problems.

[0015] SUMMARY

[0016] This summary is provided to introduce a selection of features and concepts of the invention that are further described below in the description. This summary is not intended to identify key or essential features of the claimed subject-matter, nor is it intended to be used in limiting the scope of the claimed subject-matter.

[0017] The one or more problems mentioned above are solved by the subject-matter of the independent claims. Specifically, such one or more problems are solved by a System-on-Chip Programmable Logic device (SoC PL), e.g. as part of an (mobile) edge device for example within IIoT applications, for performing a distributed and diverse inference task. The SoC PL comprises a Host Processing System (PS). The PS is configured to receive an external request. The external request relates to an inference task or includes an inference task. The inference task is accompanied by input data. This means that the external request can be followed by at least one of the inference tasks and the input data on which inference is to be performed based on an inference policy or prerequisite defined by the inference task. For example, the inference policy or prerequisite can pre-define a number of layers to be used by a deep neural network (DNN). The PS is configured to load configuration data of the DNN corresponding to the received inference task upon receiving the external request. The configuration data can include parameters of the DNN, e.g. weights and / or biases. The SoC PL comprises a Programmable Logic (PL), e.g., a Field Programmable Gate Array (FPGA). The PL comprises an input buffer. The input buffer is configured to store the loaded configuration data from the PS under control of the PS. In other words, the loaded configuration data are buffered on demand as per the request. The PL comprises a plurality of processing elements (PEs). A number of PEs can be more than 10 (or 50 or 100). The PEs are configured to perform inference on the input data under use of the DNN when the PS indicates that the input buffer is ready. This means that the inference is initiated by the PS. Further, the PEs form a systolic array.

[0018] This has the advantage that latency and power consumption requirements can be easily met.

[0019] Particularly advantageous configurations can be found in the dependent claims.

[0020] The SoC PL can further comprise a (i.e. detachable) volatile memory. The volatile memory may be associated with the PS. Further the volatile memory may be controlled by the PS. The volatile memory may be accessible by the PL via a direct memory access (DMA) arbiter. From a structural view, the DMA arbiter may be located between (adjacent to) the PL and the PS. Consequently, a simple but cost effective SoC can be provided.

[0021] The SoC PL may consist of two processing building blocks which are formed by the PL functioning as a core for computing the inference and the PS functioning at least as host central processing unit (CPU). Further the SoC PL may consist of three logical building blocks formed by the PL functioning as the core for computing the inference, the DMA arbiter functioning as access controller to the volatile memory of the PS and the PS functioning at least as the CPU.

[0022] Thus, a much simpler design for an SoC can be implemented.

[0023] The PS may be configured to load the configuration data from the volatile memory of the PS. The PS may be configured to transfer the configuration data via the DMA arbiter to the input buffer of the PL. The input buffer may be a configuration buffer. For example, the configuration buffer is only flushed by way of a (flush) command by the PS. This command can be sent via the DMA arbiter or directly to the PL. A low overhead finite state machine (FSM) may be part of the PL and configured to control all the signaling within the PL. The FSM may receive the flush command from the PS directly and then initiate processing (performing inference) by the PEs based on the flushed configuration data from the input buffer, which are received through DMA from volatile memory. The PL may be configured to store an inference result from the inference performed by the PEs in an output buffer being an activation buffer of the PL. The storing of the inference result may be done automatically in a batch-wise and / or pipeline manner into the output buffer. The output buffer may be pre-configured to indicate to the FSM that inference result is fully stored (= inference result is ready to be sent). In response, the FSM may be configured to initiate transferring the inference result directly to the PS or indirectly via the DMA arbiter to the PS. Simultaneously the FSM initiates refraining from any further processing other than transferring the inference result within the PL. The PS may be configured to fetch the inference result from the output buffer to finish the task of the SoC PL. For example, the fetched inference result may only be processed by way of the CPU of the PS, and for example circumvent the volatile memory of the PS. This can enable efficient processing at the SoC PL.

[0024] The configuration data is processed by the PL via a set sliding window. The input data is processed by the PL via the set sliding window. The sliding window may be set such that it has a fixed size. The sliding window may have a size which is configurable at a synthesize stage of an application programming interface (API) as described below. The API may provide synthesize and runtime stages for programming the PL of the SoC PL. The sliding window may be adjustable to efficiently process the configuration data with the input data. The configuration data and / or the input data have the form of a single vector or two-dimensional matrix. For example, the weights and / or the biases are respectively provided as single vector or matrix. For example, the weights are multiplicated with the input data and the biases are added thereto. For example, in batch inference, matrix-vector multiplications are iteratively executed to improve the throughput, i.e., the number of data samples processed per unit of time, with the cost of proportionally more resource usage. The Multiplication and Accumulation operation can be described such that weights, biases, and the input data are received at the PEs via the fixed-size sliding window traversing the weight matrix horizontally and vertically. After calculating the products, an adder tree may be employed followed by adding respective biases, applying the activation function, and routing to the final output buffer of the PL.

[0025] This means that the PL can be simply adjusted to corresponding latency demands of the IIoT applications.

[0026] The PS can have a device driver. The device driver may include instructions which, when executed by the PS, cause the PS to manage external data transmission, such as receiving the external request which relates to the inference task and is accompanied by the input data and / or receiving the configuration data to prepare the configuration data in the volatile memory of the PS for the loading therefrom.

[0027] The device driver may include instructions which, when executed by the PS, cause the PS to manage internal data transmission, such as input and output, between PL and the volatile memory, and PL managed by the FSM.

[0028] This makes the device driver an ideal low-level control for all the SoC PL, enabling better performance and functionality compared to an operating system (OS) which is usually part of the edge device as such.

[0029] The volatile memory may store a resumption protocol for allowing continuation of the inference task on at least one mobile edge device external to the SoC PL. Thereby, a distributed inference approach can be realized across multiple edge devices. The PS may be configured to transfer the resumption protocol to the at least one mobile edge device external to the SoC PL in association with the inference task (e.g. via the bus and the communication interface(s) of the edge device as described below). This may be done by way of broadcasting or device-to-device (D2D) with multi-hop. The resumption protocol may be forward-adjustable by the other edge device(s), such that the resumption protocol may be handed over to the other edge device(s) with an adjusted indication where to start resuming the inference task to be performed.

[0030] This enables cost-effective inference over multiple edge devices.

[0031] The SoC PL may be configured to perform the inference task up to a first split point, for example when it is the edge device that is first to perform in D2D or broadcast scenario. The SoC PL may be configured to continue the inference task from the first split point up to a second split point, for example when it is the edge device that is second (subsequent / following or even further) to perform in D2D or broadcast scenario. The SoC PL may be configured to output the inference result associated with the first and second split points as an intermediate inference result to enable another (mobile) edge device to continue and finish the inference task, e.g. per broadcast or D2D. As an example, the edge device(s) may have correspondingly different first and second split points which are predefined by the resumption protocol. Further, these first and second split points may be adjustable by the other edge device(s) which precede the edge device in a multi-hop scenario. Also, these first and second split points may be adjustable by the edge device which precede the other edge device(s) in this multi-hop scenario. Consequently, a low overhead high-speed D2D processing can be provided by the SoC PL herein.

[0032] The above-mentioned one or more problems are also solved by a computer- implemented method. The method is for synthesizing and running a System-on-Chip Programmable Logic device (SoC PL), i.e. the SoC PL as described above as part of an edge device, for performing a distributed and diverse inference task. The method comprises providing an Application Programming Interface (API). The API is for generating a Hardware Description Language (HDL). The HDL is for generating a device driver code. The device driver code is for a Host Processing System (PS). The device driver code is also for a Programmable Logic (PL). The API provides a two-stage configuration process. The two-stage configuration process comprises a synthesize stage. The synthesize stage comprises pre-configuring a deep neural network (DNN) including the number and types of layers of the DNN, and non-linearities such as ReLU, Leaky ReLU, Tanh, Sigmoid and / or Softmax. The two-stage configuration process comprises a runtime stage. The runtime stage comprises updating parameters of the DNN (such as weights and biases) on-the-fly to prepare the SoC PL or the edge device to run or accommodate new (e.g. future, upcoming) distributed and diverse inference tasks.

[0033] This has the advantage of providing an easy way to enable IIoT edge device for application.

[0034] The synthesize stage may further comprise setting interconnections between PL and PS. The synthesize stage may further comprise adjusting latency and throughput tradeoffs by changing one or more window sizes for respective one or more multiplication and accumulation operations (see set window size above). The synthesize stage may further comprise configuring minimum tolerable accuracy for various applications by setting floating-point representation, mantissa, and exponent numbers based on IEEE 784 single and half precision floating-point format. The synthesize stage may further comprise configuring minimum tolerable accuracy versus latency for various application by setting fixed-point representation. The synthesize stage may further comprise configuring the PL to run the inference with one or more preset split points, enabling distributed / split inference based on industrial internet of things, IIOT, application requirements. The synthesize stage may further comprise configuring a batch size to balance resource consumption and throughput improvement.

[0035] The runtime stage may further comprise managing input transmission to the PL and output reception from the PL (with minimal latency). The runtime stage may further comprise configuring one or more preset split points (e.g. the first and / or second split points) for distributed / split inference of the inference task, enabling inference up to a layer in the DNN specified by the one or more preset split points (e.g. the first split point or the second split point). The runtime stage may further comprise implementing an early exit point (e.g. the first split point) for scenario adaptation based on latency requirements or residual power budget of the SoC PL. The runtime stage may further comprise providing a runtime status monitoring tool (e.g. as part of the communication interface(s) as described below) displaying power consumption, latency, throughput, and / or the accuracy of a current inference result.

[0036] The synthesize stage may further comprise pre-configuring the PS, such that the PS receives an external request which relates to the inference task and is accompanied by input data. The synthesize stage may further comprise pre-configuring the PS, such that the PS loads configuration data of the DNN corresponding to the received inference task upon receiving the external request. The synthesize stage may further comprise preconfiguring the PL, such that an input buffer of the PL stores the loaded configuration data from the PS under control of the PS. The synthesize stage may further comprise preconfiguring the PL, such that a plurality of processing elements (PEs) of the PL performs inference on the input data under use of the DNN when the PS indicates that the input buffer is ready. The synthesize stage may further comprise pre-configuring the PL, such that pre-configuring the PEs in form of a systolic array.

[0037] Pre-configuration of the SoC PL may increase reliability during runtime.

[0038] The synthesize stage may enable the SoC PL to be tailored to a specific industrial internet of things, IIOT, application, resulting in optimized performance and efficiency. The runtime stage may provide dynamic adaptability to changing operational conditions and task requirements in real-time.

[0039] The above-mentioned one or more problems are also solved by a computer- implemented method of operating a System-on-Chip Programmable Logic device (SoC PL) for performing a distributed and diverse inference task. The method comprises receiving an external request which relates to the inference task, and is accompanied by input data. The method comprises loading configuration data of a deep neural network (DNN) corresponding to the received inference task upon receiving the external request. The method comprises storing the loaded configuration data from a Host Processing System (PS) under control of the PS. The method comprises performing, at a plurality of processing elements (PEs) of a Programmable Logic (PL) inference on the input data under use of the DNN when the PS indicates that an input buffer of the PL is ready. The PEs form a systolic array.

[0040] The above-mentioned one or more problems are also solved by a computer program. The computer program comprises instructions which, when the computer program is executed by a computer, cause the computer to carry out or initiate the method as described above or at least one of the steps thereof. The computer program may be, for example, a module for starting / operating the SoC PL, the PS and / or the PL as described herein.

[0041] The above-mentioned one or more problems are also solved by a data carrier. The computer program may be stored on the machine-, processor- or computer- readable data carrier, such as a permanent or rewritable storage medium. This also encompasses that the computer program may be made available for download on a server or cloud server, such as over a data network such as the Internet or a communication link such as a wireless connection.

[0042] In other words, the invention concerns a method and device to facilitate the implementation of embedded machine learning algorithms on system on chip programmable logic (SoC PL), such as FPGAs. The resulting implementation can support the effective adaptation of the 6th Generation (6G) wireless systems to changes in the signal propagation environments under stringent energy consumption and ultra-low operational latency. This may be for ensuring that 6G systems can dynamically respond to environmental changes efficiently and swiftly, meeting the high-performance demands of modern wireless communication standards. Furthermore, since the implementation of a machine learning algorithm on multiple FPGA-supported (in HDL) devices demands considerable time and signal synchronization efforts, the method and device may provide efficient implementation based on integrated higher-level programming API design. Thereby, the need for repetitive development in HDL and reprogramming of SoC PL can be significantly reduced. A dedicated analyze platform may be envisaged herein, empowering users to thoroughly test and debug the design before its final implementation.

[0043] One of the problems concerned herein may be the efficient and reprogrammable usage of system on chip programmable logic resources for distributed machine learning, i.e., training and inference, in 6G wireless systems. The method and device herein may provide a fast and reconfigurable implementation of different machine learning algorithms while optimizing critical factors such as energy consumption, operational latency, and throughput (operations per unit of time). The method and device herein may accelerate the implementation of distributed machine learning on SoC PLs, after which the machine learning model can be seamlessly updated through a host processing unit while the system is fully operational in the environment.

[0044] In the context of 6G distributed systems, splitting the DNN may be beneficial. It can allow the system to handle a variety of tasks that have different requirements. For instance, some tasks may be latency-intensive, meaning they require a quick response. Other tasks may be throughput-intensive, meaning they involve processing large amounts of data. The method and device herein may enable splitting the DNN, enabling the system to more effectively address these diverse needs.

[0045] At least one or more of the following aspects may be addressed by the method and device herein. Implementation of deep reinforcement learning on SoC PL devices. Integration of reinforcement learning in efficient and low-latency hardware for real-time optimization, decision-making, and adapting to dynamic environments with ultra-low latency requirements. Automatic splitting of the DNN, as required for the implementation of distributed learning, based on latency and throughput requirements. Splitting the DNN for distributed systems in 6G may help run inference to address heterogeneous tasks, such as latency-intensive and throughput-intensive tasks. Modifying DNN parameters on-the- fly while monitoring the performance of the DNN for both analysis and design purposes. Dynamic adjustment allowing for real-time optimization and fine-tuning, enhancing the model's adaptability and performance in varying conditions. Implementing on-the-fly parameter updates for improving responsiveness and effectiveness of DNNs in real -world applications. Achieving operational latency below 0.1 ms in 6G wireless applications. No need for reprogramming SoC PL over-the-air. Over-the-air machine learning model structure updates as a requirement in 6G wireless systems. Flexibility and a trade-off between operational latency and throughput, despite the changes in the environment. Incorporation of adaptive updates of the machine learning parameters over the air. This aspect may be fundamentally crucial for highly dynamic 6G wireless applications such as mobile robotics in a production hall, as they necessitate reconfiguration based on new objectives and physical changes in the environment.

[0046] At least one or more of the following advantages may be achieved correspondingly. A programmable API may be provided for efficient configuration of the machine learning architecture combined with the SoC PL logic reconfiguration. A method and architecture that eliminates the need for local data reuse based on systolic arrays may decrease the operational latency in the order of below 0.1ms. A Direct Memory Access (DMA) with a parameter fetching mechanism may enable seamless over-the-air reconfigurability. This can allow the SoC PL to retrieve updated parameters seamlessly, ensuring ultra-low latency. A heuristic algorithm can automatically split the DNN and deploy it across devices, meeting the latency and power consumption requirements of large-scale distributed systems. By utilizing a host processor with high-speed communication to the system-on-chip programmable logic, our method can enable the implementation of reinforcement learning to adapt to dynamic systems. Our innovation accelerates the implementation of efficient machine learning algorithms on multiple SoC PLs simultaneously. This will help engineers validate and speed up the testing of both centralized and more challenging distributed algorithms in real-world implementations. Our method will complement the computer simulationbased testing and validation of algorithms. Consequently, this approach adds significant value to the evaluation of various machine learning algorithms, as they are executed in hardware. Importantly, our method eliminates the need for complex HDL code execution and validation, making SoC PL implementation accessible through standard APIs. This democratization of SoC PL utilization enhances the accessibility and applicability of machine learning algorithms in hardware, fostering a more inclusive and practical approach to algorithm validation.

[0047] Although some of the aspects described above relate to the method(s), the edge device or the SoC PL, these aspects may also apply in a corresponding manner to the respective other aspects thereof.

[0048] In an example, the edge device / SoC PL may be implemented under use of hardware circuits, software means or a combination thereof. Thus, several components of the edge device / SoC PL can each be implemented in a single physical entity, for example if several functions are implemented in software. The components of the edge device / SoC PL can also be implemented in hardware modules. The components of the edge device / SoC PL are each to be understood as functional units that are not necessarily physically separated from one another. That is, the edge device / SoC PL may be implemented partially as computer, field programmable logic array (FPLA), field programmable gate array (FPGA), microcontroller, CPU (e.g., multi-core processor), Graphics Processing Unit (GPU), Application Specific Integrated Circuit (ASIC) and / or Digital Signal Processor (DSP).

[0049] In the edge device / SoC PL, for example, procedures can be used that are related to pipelining of task-related data meaning exactly one or more of: the inference task, the configuration data, the external request and the input data. Here, instead of an entire instruction in one clock cycle of the PS and / or PL of the edge device / SoC PL, only a subtask thereof, e.g. a part of the task-related data, is processed. The various subtasks of several instructions may be processed simultaneously. Furthermore, procedures in the sense of multithreading can be applied to the task-related data and further developments thereof, for example simultaneous multithreading of the task-related data. This makes it possible to achieve better utilization of the PS and / or PL due to parallel use of several processor cores. Here, the edge device / SoC PL can be scalar or superscalar. The PS and / or PL contained in the edge device / SoC PL can be connected to a buffer memory of the edge device / SoC PL, which can temporarily store the task-related data before and / or after the processing of the task-related data or the part thereof (according to the subtask). The buffer memory may be integrated in a volatile memory device of the edge device / SoC PL, e.g. a (D)RAM, or a non-volatile memory device of the edge device / SoC PL, e.g. an SSD. This may increase a performance of the edge device / SoC PL.

[0050] All technical and scientific terms used herein have the meaning which corresponds to the general understanding of the skilled person in the technical field of edge computing; they are to be understood based on the definition in a lexicon or technical jargon about this technical field. If technical terms are used incorrectly herein, and thus do not reflect the technical concept of the present invention, these may be replaced by technical terms which convey a correct understanding to the skilled person.

[0051] With the terms "first", "second", etc. elements should be distinguished from one another. For example, a first element may be referred to as a second element and a second element may also be referred to as a first element. It should be noted that these terms as well as all numerical indications ("one", "two", etc.) are to be understood as open for the scope of protection, but are also co-disclosed as closed for the disclosure content. For example, the expression "two ABCs" can mean either "exactly two ABCs" or "two or more ABCs".

[0052] In the present disclosure, if a component is "connected to” or “in communication with” another component, this can mean that these components can also be directly connected or communicate with each other. The term "directly" indicates that there is no other component in between.

[0053] The steps of the method(s) disclosed herein may not be construed as to be within a specific order, unless explicitly or implicitly stated otherwise, e.g., that such steps are not interchangeable for technical reasons. Also, the steps may be performed directly one after the other (with no other steps in between) and / or continuously.

[0054] BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other objects, features, advantages and applications will become apparent from the following description of non-limiting embodiments with reference to the accompanying drawings. The same or similar components are always provided with the same or similar reference signs. Detailed explanations of known functions or structures are omitted, insofar as they are unnecessarily distracting from the invention.

[0056] The drawings show in:

[0057] Fig. 1 an illustration of an edge device embedding an SoC PL;

[0058] Fig. 2 an illustration of a method for synthesizing and running the SoC PL; and

[0059] Fig. 3 an illustration of a method of operating the SoC PL.

[0060] DETAILED DESCRIPTION OF THE DRAWINGS

[0061] The edge device, the SoC PL and the computer-implemented methods will now be described with respect to the embodiments. In particular, without being restricted thereto, specific details are set forth to provide a thorough understanding of the invention.

[0062] Fig. 1 shows an illustration of an edge device 1 embedding an SoC PL 2. Specifically, Fig. 1 schematically shows a block diagram representing an edge device 1 and in particular an SoC PL 2. The SoC PL 2 can represent at least a part of the edge device 1 or be an integrated part thereof. Thus, the aspects relating to the edge device 1 in the following may apply to the SoC PL 2 itself. The edge device 1 is implemented by way of one or more steps of the method S20 as represented in Fig. 2. In particular, the edge device 1 provides functionality, such as computer software, which runs on the edge device 1 and performs step S23 of the method S20 and / or one or more steps of the method S30 as represented in Fig. 3.

[0063] For the ease of understanding, the term task-related data used in the following may mean one or more of the following data or information mentioned herein: the inference task, the configuration data, the external request, the input data.

[0064] In particular, the edge device 1 may execute instructions associated with the task- related data included in the computer program described herein and cause the edge device 1 to perform or initiate one or more of the steps of the method S30.

[0065] It is provided herein that the edge device 1 takes any suitable physical form. By way of example, the edge device 1 may be at least partially adapted to be an embedded computer, system-on-chip (SoC), single board computer (SBC), server and / or user equipment (UE). The edge device 1 may be unitary or distributed; span one or more locations or one or more machines / compute centers. The edge device 1 may perform or initiate step S23 of the method S20 and / or one or more steps of the method S30 without significant spatial or temporal limitations. As an example, the edge device 1 may perform or initiate step S23 of the method S20 and / or one or more steps of the method S30 in real time, in parallel or in batch mode. The edge device 1 may perform or initiate step S23 of the method S20 and one or more steps of the method S30 at different times or locations.

[0066] The edge device 1 has at least one or more of the following components: SoC PL 2, PS 3, volatile memory 4, CPU 5, DMA arbiter 6, PL 7, FSM 8, input buffer 9, output buffer 10, PEs 11, bus 12, communication interface(s) 13, persistent storage 14, main power supply 15 and auxiliary power supply 16. The components of the edge device 1 may be at least partially performed in hardware and / or software. The interconnection of the components of the edge device 1 is structured as in Fig. 1 merely for the sake of simplicity. In particular, the interconnection and connection may differ in the implementation due to signal processing and signaling.

[0067] The PS 3 has means for performing instructions associated with the task-related data, e.g., the computer program described herein. For example, the PS 3 may load the instructions associated with the task-related data included in the computer program described herein, e.g., from the volatile memory 4 and / or the persistent storage 14, and then execute the instructions, which in turn causes the PS 3 to perform or initiate step S23 of the method S20 as represented, for example, in Fig. 2, and / or to perform or initiate one or more steps of the method S30 as represented, for example, in Fig. 3. The PS 3 may have an internal register / cache for the task-related data, for instructions associated with the task-related data, and / or for associated addresses. The PS 3 may therefor use the CPU 5 for accessing the internal register / cache. As an example, to perform the instructions associated with the task-related data, the PS 3 may fetch them from the internal register / cache of the PS 3, the volatile memory 4, or the persistent storage 14; decode and perform them; and then write a result to the internal register / cache of the PS 3, the volatile memory 4, or the persistent storage 14.

[0068] As an example, the PS 3 may have an instruction cache, a data cache, and / or a translation buffer (TLB). The instructions associated with the task-related data in the instruction cache may be copies of instructions in the volatile memory 4 and / or persistent storage 14, and the instruction cache may accelerate fetching of these instructions associated with the task-related data by the PS 3. The task-related data in the data cache may be copies of data for the instructions currently executing on the PS 3 and related to the task-related data in the volatile memory 4 and / or persistent storage 14. The results of the previous instructions performed on the PS 3 and related to the task-related data, may be provided for access by subsequent instructions to be performed on the PS 3 and related to the task-related data, or for writing to the volatile memory 4 and / or persistent storage 14. The data cache may accelerate the read or write operations of the PS 3. The addresses in the TLB associated with the task-related data may be address references to addresses in the volatile memory 4 and / or persistent storage 14 to accelerate virtual address translation for the PS 3.

[0069] The volatile memory 4 can be a dynamic RAM (DRAM) or a static RAM (SRAM). In particular, the volatile memory 4 may be adapted to be the data carrier described herein on which the computer program described herein may be at least temporarily stored. In addition, the volatile memory 4 may be a single or multi-channel RAM. The volatile memory 4 may have a main memory for storing instructions associated with the task- related data for the PS 3, which then performs those instructions; or may have the task- related data for the PS 3 to use to operate on. As an example, the edge device 1 may load these instructions from the persistent storage 14 or another source (such as another computer, the network, or the cloud) into the volatile memory 4. The PS 3 may then load these instructions from the volatile memory 4 into the internal register / cache of the PS 3. To perform these instructions, the PS 3 may fetch and decode these instructions from the corresponding internal register / cache. During or after performing these instructions, the PS 3 may write a result (which may be intermediate or final results) to the internal register / cache. The PS 3 may then write the result to the volatile memory 4.

[0070] For example, the PS 3 performs only the instructions related to the task-related data in the internal register / cache of the PS 3 or in the volatile memory 4 (as opposed to the persistent storage 14), and operates only on the task-related data in the internal register / cache of the PS 3 or in the volatile memory 4 (as opposed to the persistent storage 14).

[0071] The persistent storage 14 has a non-volatile (mass) memory (NVM) for the task- related data or the instructions associated with the task-related data. In particular, the persistent storage 14 may be adapted to be the data carrier described herein, on which the computer program described herein may be stored. As an example, the persistent storage 14 can be a flash memory, in particular SSD or eMMC. The persistent storage 14 can store the task-related data in an erasable or non-erasable manner. The persistent storage 14 can be located in the edge device 1, i.e. internally, or connected externally.

[0072] The PS 3 can be connected directly or indirectly, e.g. via the DMA arbiter 6, to the volatile memory 4 and / or persistent storage 14 by way of e.g. the bus 12 as illustrated in Fig. 1. The connection can be made via a clock bus, command bus and data bus. The persistent storage 14 can also be part of the SoC PL 2. The volatile memory 4 and / or persistent storage 14 receives commands associated with the task-related data and the task-related data in connection with a clock signal that is specified by the PS 3 on the clock bus. Here, the clock signal clocks the receipt of the commands associated with the task-related data and the task-related data. The PS 3 transmits a command related to the task-related data to the volatile memory 4 and / or persistent storage 14 via the command bus. Further, the PS 3 transmits the task-related data corresponding to the command to the volatile memory 4 and / or persistent storage 14 via the data bus or receives the task- related data from the volatile memory 4 and / or persistent storage 14 via the data bus.

[0073] In one example, the volatile memory 4 and / or persistent storage 14 may have a clock pin via which the clock signal is received at the volatile memory 4 and / or persistent storage 14. The clock signal may be a write enable signal and / or read enable signal. The volatile memory 4 and / or persistent storage 14 may further have a first and second input / output (I / O) pin. The task-related data is received at the volatile memory 4 and / or persistent storage 14 in synchronization with the clock signal via the first I / O pin. The volatile memory 4 and / or persistent storage 14 can further have a command / address buffer, a control logic and an I / O buffer. The command / address buffer operates at a first operating speed and buffers the command and corresponding address received via the second I / O pin and related to the task-related data in synchronization with the clock signal. The I / O buffer operates at the first operating speed and buffers the task-related data as read data from the volatile memory 4 and / or persistent storage 14 or writes the task-related data as write data to the volatile memory 4 and / or persistent storage 14. The first and second I / O pin can coincide, i.e. be the same pin. Here, the clock signal may be formed by a first and second clock signal in which the first clock signal switches only in a period in which the command and the address (both related to the task-related data) are received from the volatile memory 4 and / or persistent storage 14, and the second clock signal switches only in a period in which the task-related data is received from the volatile memory 4 and / or persistent storage 14. The first operating speed corresponds to a data input speed or data output speed between the volatile memory 4 and / or persistent storage 14 and the PS 3. The control logic controls an operation with respect to the volatile memory 4 and / or persistent storage 14 based on the buffered instruction and the buffered address (both related to the task-related data). Here, the control logic operates at a second operating speed that is lower than the first operating speed and corresponds to an internal operating speed of the volatile memory 4 and / or persistent storage 14.

[0074] The bus 12 may be understood herein as a subsystem of the edge device 1 and / or the SoC PL 2, which transmits the task-related data and / or electrical power between the components of the edge device 1. The (one) bus 12 may thereby interconnect the components of the edge device 1 via the same set of wires. The bus 12 may be adapted to provide dedicated communication of the task-related data between two or more of the components of the edge device 1. The bus 12 may be a system bus through which the PS 3 is connected to the other components of the edge device 1. Here, the bus 12 can be synchronous - the transfer of the task-related data takes place bidirectionally with a clock edge of a clocking of the bus 12 - and / or asynchronous - no clocking, but a handshake takes place for the transfer of the task-related data. In such a semi -synchronous system bus, the bus 12 is clocked, but control lines enable wait cycles in order to also use slow components such as the persistent storage 14 via the bus 12.

[0075] The DMA arbiter 6 may be provided for at least partial control over data access towards the volatile memory 4 via e.g. the bus 12. The DMA arbiter 6 may be understood as a coprocessor (ancillary) to the PS 3. Based on a two-way handshake or three-way handshake, the DMA arbiter 6 controls the access (to the bus 12, e.g. for accessing the volatile memory 4 of the PS 3) associated with the task-related data. The three signals bus request (BREQ) for forwarding the task-related data, bus grant (BGRT) for confirming and approving the forwarding and bus grant acknowledge (BGA) for optional forwarding feedback are used for this purpose. The DMA arbiter 6 receives several BREQs simultaneously from different components of the edge device 1, in particular the SoC PL 2, e.g. via the bus 12. The DMA arbiter 6 sorts the BREQs according to priority and forwards them sequentially - in a pipeline - to the PS 3. As soon as the PS 3 has received the BREQ, the PS 3 transmits the BGRT to the DMA arbiter 6 or directly to the component of edge device 1 and / or SoC PL 2 transmitting the BREQ, wherein the component particularly means the PL 7 herein. A lower-priority BREQ of the BREQs in the pipeline - e.g., from another component of the edge device 1 - is forwarded to the PS 3 in response to a BGRT sent by the PS 3 with respect to the BREQ having priority in the pipeline and related to at least a portion of the task-related data. The BGRT related to the lower priority BREQ is transmitted by the PS 3 to the DMA arbiter 6 after processing the at least part of the task-related data. The DMA arbiter 6 may, for example, in response to the BGRT relating to the lower-ranking BREQ, in turn transmit a further lower-ranking BREQ in the pipeline - relating, for example, to another portion of the task-related data - of the BREQs to the PS 3. Similarly, in response to each BGRT from the PS 3, the DMA arbiter 6 may transmit a respective related BGA to the PS 3. With the procedure described here, a BGA can also be omitted entirely. This saves overhead in the communication between the components of the edge device 1, especially the SoC PL 2. In other words, a two-way handshake is provided instead of a three-way handshake.

[0076] The bus 12 can also have a data bus, address bus (command bus) and control bus. In this case, the task-related data is transmitted bidirectionally between the components of the edge device 1 and / or the SoC PL 2 via the data bus. The address bus is operated solely by the PS 3 and transmits memory addresses related to the task-related data unidirectionally. The control bus is controlled solely by the DMA arbiter 6, e.g. in the sense of a monitor, and transfer control thereof to the PS 3 in the pipeline-like manner as described above to control the transmission of the task-related data.

[0077] The communication interface(s) 13 enable the edge device 1 to communicate with a network, preferably with a network comprising other edge devices 1 (not shown in Fig. 1) or an access node for transferring task-related data to the other edge devices 1. For example, this may be an ad hoc network, wireless personal area network ((W)PAN), e.g. a Bluetooth WPAN, local area network (LAN), and / or WI-FI network comprising the other edge devices 1. Alternatively or additionally, a direct communication link with the other edge devices 1 may be established via the communication interface(s) 13, e.g. a D2D communication, especially via a peer-to-peer communication (P2P) with multi-hop. This enables a transmission that is less burdened by traffic. The communication interface(s) 13 enable the edge device 1 to communicate with a network, such as Bluetooth, WLAN, another edge device 1 via D2D in P2P with multihop, cellular system and / or at least a portion of the Internet, when the edge device 1 is in a connected mode. When in a disconnected mode, the edge device 1 may stop communicating and / or powering wireless interface(s) other than a GNSS receiver of the communication interface(s) 13.

[0078] The communication interface(s) 13 may enable user interaction with the edge device 1 in an environment of the edge device 1 (for example, user commands in the environment of the edge device 1). The communication interface(s) 13 may have device and / or software drivers that enable the PS 3 to control the communication interface(s) 13 to communicate an action instruction based on the task-related data to the SoC PL 2, in particular to the PS 3 as host controller of the SoC PL 2 as a general controller system of the edge device 1.

[0079] The communication interface(s) 13 provide means for communicating (such as packet-based communication) the task-related data between the edge device 1 and other communication participants, preferably the other edge devices 1, e.g. via a network connecting all edge devices 1, or an access node / base station (gNB) (wired - via cable, e.g. optical connection - and / or wireless - via antenna).

[0080] The edge device 1 as an (NR) UE can be used in Narrow Band (NB) Internet of Things (loT) applications where only occasional and small amounts of data are transmitted in the uplink (UL), such as the task- related data. For example, the task-related data may be transmitted when the edge device 1 is in a radio resource control (RRC CONNECTED) state that requires a significant amount of electrical power from the main power supply 15 or the auxiliary power supply 16. However, since the amount of NB-IOT data is small, the task-related data can be transmitted less frequently and more efficiently. In particular, the different RRC states of the edge device 1 consume different amounts of resources, and therefore the transition between the RRC states may efficiently reduce network resources. The edge device 1 may be in one of the following states: (NR) RRC CONNECTED state, (NR) RRC INACTIVE state and (NR) RRC IDLE state. If the edge device 1 is switched off (e.g. if no electrical power is supplied by the main power supply 15 and / or the auxiliary power supply 16), the edge device 1 is in a disconnected state and is not in any of the three RRC states. After switching on the edge device 1, the edge device 1 can initially enter the RRC IDLE state. In the RRC IDLE state, the edge device 1 can attempt to establish a wireless connection with an operating base station (e.g. gNB - not shown) and transition to the RRC CONNECTED state. After the edge device 1 transitions, the edge device 1 can also be released from the RRC CONNECTED state to return to the RRC IDLE state. However, after the initial transition to the RRC CONNECTED state, the edge device 1 can transition to the RRC INACTIVE state in order to use network resources more efficiently. The RRC INACTIVE state of the edge device 1 can be released, resumed or suspended in order to transition back to the RRC CONNECTED state. In addition, the edge device 1 can be released from the RRC INACTIVE state and return to the RRC IDLE state. The RRC INACTIVE state minimizes the latency time and reduces the signaling load, which means that the network resources are used more efficiently and the power consumption of the edge device 1 is reduced when transmitting / receiving the task-related data.

[0081] For example, the edge device 1 may be part of a 4-stage random access channel (RACH) transmission method that includes transmitting four messages (Msgl, Msg2, Msg3, and Msg4) prior to transmitting the task-related data to the serving base station. Here, the edge device 1 may perform random access by having the edge device 1 transmit a RACH preamble - e.g., Msgl - on a RACH resource. The serving base station can respond with a Random Access Response (RAR) - e.g. Msg2. The edge device 1 may then transmit a Radio Resource Control (RRC) connection request - e.g. Msg3 - in the Physical Uplink Shared Channel (PUSCH) (e.g. NR-PUSCH). The serving base station can then respond with an RRC Connection Setup - e.g. Msg4 - which completes the initial access process of the edge device 1. This RACH mode is an inefficient way of transmitting the task-related data, as the task-related data is only transmitted between the edge device 1 and the operating base station after the four messages.

[0082] As another example, the edge device 1 may be part of an Early Data Transmission (EDT) method that includes transmitting two messages (Msgl and Msg2) prior to transmitting the task-related data to the serving base station. That is, in this EDT mode, the edge device 1 can transmit the task-related data in message 3 (Msg3) and the serving base station can transmit the downlink (DL) data in message 4 (Msg4) of the (legacy) 4- step RACH mode. This way of transmitting the task-related data is more efficient than the 4-step RACH mode. The edge device 1 can continue to transmit / receive UL / DL data packets in EDT mode after Msg4 in RRC IDLE state or RRC INACTIVE state.

[0083] The edge device 1 can also broadcast / transmit the task-related data for a predefined event via the communication interface(s) 13. This predefined event may be a termination of the successful establishment of a communication connection with a network. In this case, the edge device 1 can retrieve the task-related data stored in the volatile memory 4 and forward it in packets via the communication interface(s) 13 over the cellular system.

[0084] In the event of a handover, for example when the motor vehicle including the edge device 1 switches between two cells of the cellular system, the task-related data may be held in packets in the edge device 1 such that a first part of the packets is handed over in a handover cell and a second part of the packets is handed over in a service cell including the serving base station. Ultimately, the task-related data consisting of the first and second parts of the packets are thus assembled in the (core) network underlying the cellular system itself.

[0085] The main power supply 15 supplies electrical power to at least one or more of the components of the edge device 1, for example via the bus 12. In particular, the main power supply 15 charges the auxiliary power supply 16 with electrical power, for example from outside the edge device 1, e.g. in the case where the main power supply 15 is connected to the power source outside the edge device 1. Here, the main power supply 15 may represent a preferred component used for powering the components of the edge device 1 and may have, for example, an accumulator or a battery. The main power supply 15 may have other components such as voltage regulators, DC voltage stabilizers, series regulators, buck converters, and / or boost converters to meet the corresponding requirements of the components of the edge device 1. Here, the main power supply 15 may have either a dedicated fixed power supply connection with the external power source such as a electrical power supply system or a detachable power supply connection for charging the accumulator or battery of the main power supply 15. For this purpose, the main power supply 15 may have an inverter to provide a predetermined DC power supply from a connected AC power source as the external power source. The predetermined DC power supply may also already be provided from a connected DC power source as the external power source. The DC power supply may be regulated by the above-mentioned voltage regulators and supplied to the components controlled by the edge device 1, or particularly the SoC PL 2, as set DC power supplies.

[0086] The auxiliary power supply 16 is connected to the volatile memory 4 and / or the persistent storage 14 via the bus 12. The auxiliary power supply 16 is charged by the electrical power of the main power supply 15. The auxiliary power supply 16 may be located inside or outside the SoC PL 2, or inside or outside the volatile memory 4 and / or the persistent storage 14. For example, the auxiliary power supply 16 may be accommodated on a motherboard of the edge device 1 or within the integrated circuit of the SoC PL 2 to provide auxiliary power to the volatile memory 4 and / or the persistent storage 14. In particular, the auxiliary power supply 16 may be implemented at least partly in the form of a supercapacitor, an accumulator and / or a battery. The power capacity / energy capacity of the main power supply 15 can be many times greater, for example at least 10 times or 50 times greater, than the power capacity / energy capacity of the auxiliary power supply 16.

[0087] The PS 3 monitors changes in the electrical power supplied by the main power supply 15. In the event of a sudden power failure, such as when the power source external to the edge device 1 is disconnected from the main power supply 15 or the main power supply 15 degrades or fails for some other reason, and the PS 3 determines that the electrical power supplied by the main power supply 15 to one or more of the components of the edge device 1, in particular the SoC PL 2 components, has fallen below a threshold value, such as 0.8 or 0.75 of an operating power of the main power supply 15, the PS 3 causes the auxiliary power supply 16 to take over a remaining supply power for a shutdown operation of the edge device 1. The shutdown operation comprises supplying power to at least the PS 3, the volatile memory 4 and / or the persistent storage 14 with electrical power for the time of the shutdown operation. During the shutdown process, the task-related data currently located in the volatile memory 4 and / or the task-related data currently being processed in the PS 3, for example in the register / cache of the PS 3, are transferred from the volatile memory 4 and / or the PS 3 to a meta area of the persistent storage 14. For this purpose, the meta area of the persistent storage 14 can be reserved especially for the shutdown process.

[0088] The PS 3, in the case of a startup operation of the edge device 1 in which the main power supply 15 again provides the operating power, loads the task-related data from the meta area of the persistent storage 14 to enable faster data processing. After the startup process, the meta area of the persistent storage 14 can be released or successively during the startup process.

[0089] The edge device 1, in particular the SoC PL 2, may be used in the computer- implemented methods S20 in Fig. 2 and S30 in Fig. 3 as follows.

[0090] Fig. 2 shows an illustration of the computer-implemented method S20 for synthesizing and running the SoC PL 2. The SoC PL 2 is for performing a distributed and diverse inference task. The method S20 comprises providing at S21 the API for generating the HDL and a device driver code for the PS 3 and the PL 7, respectively. The API provides a two-stage configuration process comprising a synthesize stage at S22 comprising pre-configuring the DNN including the number and types of layers of the DNN, and non-linearities thereof, and a runtime stage at S23 comprising updating parameters of the DNN on-the-fly to prepare the SoC PL 2 or the edge device 1 to run new (future) distributed and diverse inference tasks.

[0091] Fig. 3 shows an illustration of the computer-implemented method S30 of operating the SoC PL 2. The SoC PL 2 is for performing a distributed and diverse inference task. The method S30 comprises receiving at S31 an external request which relates to the inference task, and is accompanied by input data. The method S30 comprises loading at S32 configuration data of the DNN corresponding to the received inference task upon receiving the external request. The method S30 comprises storing at S33 the loaded configuration data from the PS 3 under control of the PS 3. The method S30 comprises performing at S34, at the plurality of PEs 11 of the PL 7, inference on the input data under use of the DNN when the PS 3 indicates that an input buffer 9 of the PL 7 is ready. The PEs 11 form a systolic array.

[0092] The steps S21, S22 and S23 of method S20 as well as the steps S31, S32, S33 and S34 of method S30 that are illustrated as blocks in the respective block diagram in both Fig. 2 and 3 may, for example, be substantially represented in machine-, processor- or computer-readable data carrier and so executed by a computer or processor, as described e.g. below with reference to Fig. 1 within the edge device 1. Examples may further be or relate to a computer program having a program code for performing at least part of the method steps, when the computer program is executed on the computer or processor. Examples may also cover a volatile memory 4 or persistent storage 14, as e.g. also described below with reference to Fig. 1, which is machine-, processor- or computer- readable and encodes machine-executable, processor-executable or computer-executable programs of instructions which cause performing some or all of the method steps.

[0093] At this point it should be noted that all of the above-described parts are considered to be essential to the invention on their own and in any combination, especially the details shown in the drawings. Modifications of this are familiar to the skilled person.

[0094] LIST OF REFERENCE SIGNS

[0095] 1 Edge Device

[0096] 2 System-on-Chip Programmable Logic (SoC PL)

[0097] 3 Host Processing System (PS)

[0098] 4 Volatile Memory

[0099] 5 Central Processing Unit (CPU)

[0100] 6 Direct Memory Access (DMA)

[0101] 7 Programmable Logic (PL)

[0102] 8 Finite State Machine (FSM)

[0103] 9 Input Buffer

[0104] 10 Output Buffer

[0105] 11 Processing Elements (PEs)

[0106] 12 Bus

[0107] 13 Communication Interface(s)

[0108] 14 Persistent Storage

[0109] 15 Main Power Supply

[0110] 16 Auxiliary Power Supply

Claims

CLAIMS1. A System-on-Chip Programmable Logic device, SoC PL (2), for performing a distributed and diverse inference task, the SoC PL (2) comprising: a Host Processing System, PS (3), configured to: receive an external request which relates to the inference task, and is accompanied by input data, and load configuration data of a deep neural network, DNN, corresponding to the received inference task upon receiving the external request; and a Programmable Logic, PL (7), including: an input buffer (9) configured to store the loaded configuration data from the PS (3) under control of the PS (3), and a plurality of processing elements, PEs (11), configured to perform inference on the input data under use of the DNN when the PS (3) indicates that the input buffer (9) is ready; characterized in that the PEs (11) form a systolic array.

2. The SoC PL (2) of claim 1, characterized in that the SoC PL (2) further comprises a detachable volatile memory (4) associated and controlled by the PS (3), and accessible by the PL (7) via a direct memory access, DMA, arbiter (6).

3. The SoC PL (2) of claim 1 or 2, characterized in that the SoC PL (2) consists of two processing building blocks which are formed by the PL (7) functioning as a core for computing the inference and the PS functioning as host central processing unit, CPU (5).

4. The SoC PL (2) of any one of claims 1 to 3, characterized in that the PS (3) is configured to load the configuration data from the volatile memory (4) of the PS (3) and to transfer the configuration data via the DMA arbiter (6) to the input buffer (9) of the PL (7) that is a configuration buffer, the PL (7) is configured to store an inference result from the inference performed by the PEs (11) in an output buffer (10) being an activation buffer of the PL (7), and the PS (3) is configured to fetch the inference result from the output buffer (10) to finish the task of the SoC PL (2).

5. The SoC PL (2) of any one of claims 1 to 4, characterized in that the configuration data and the input data are processed by the PL (7) via a set sliding window having a size which is configurable at a synthesize stage (S22) of an application programming interface, API.

6. The SoC PL (2) of any one of claims 1 to 5, characterized in that the PS (3) has a device driver including instructions which, when executed by the PS (3), cause the PS (3) to manage:external data transmission, such as receiving the external request which relates to the inference task and is accompanied by the input data and / or receiving the configuration data to prepare the configuration data in the volatile memory (4) of the PS (3) for the loading therefrom, and internal data transmission, such as input and output, between PL (7) and the volatile memory (4), and PL (7) managed by a low overhead finite state machine, FSM (8).

7. The SoC PL (2) of any one of claims 1 to 5, characterized in that the volatile memory (4) stores a resumption protocol for allowing continuation of the inference task on at least one edge device (1) external to the SoC PL (2); and wherein the PS (3) is configured to transfer the resumption protocol to the at least one edge device (1) external to the SoC PL (2) in association with the inference task.

8. The SoC PL (2) of claim 7, characterized in that the SoC PL (2) is configured to: perform the inference task up to a first split point; continue the inference task from the first split point up to a second split point; output the inference result associated with the first and second split points as an intermediate inference result to enable another edge device (1) to continue and finish the inference task.

9. A computer-implemented method (S20) for synthesizing and running a System-on- Chip Programmable Logic device, SoC PL (2), i.e. the SoC PL (2) of any one of claims 1 to 8 of an edge device 1, for performing a distributed and diverse inference task, the method (S20) comprising: providing (S21) an Application Programming Interface, API, for generating a Hardware Description Language, HDL, and a device driver code for a Host Processing System, PS (3), and a Programmable Logic, PL (7), respectively, characterized in that the API provides a two-stage configuration process comprising: a synthesize stage (S22) comprising pre-configuring a deep neural network, DNN, including the number and types of layers of the DNN, and nonlinearities thereof; and a runtime stage (S23) comprising updating parameters of the DNN on-the-fly to prepare edge device (1) to run new distributed and diverse inference tasks.

10. The computer-implemented method (S20) of claim 9, characterized in that the synthesize stage (S22) further comprises: setting interconnections between PL (7) and PS (3); adjusting latency and throughput trade-offs by changing one or more window sizes for respective one or more multiplication and accumulation operations; configuring minimum tolerable accuracy for various applications by setting floating-point representation, mantissa, and exponent numbers based on IEEE 784 single and half precision floating-point format;configuring the PL (7) to run the inference with one or more preset split points, enabling distributed / split inference based on industrial internet of things, IIOT, application requirements; configuring a batch size to balance resource consumption and throughput improvement.

11. The computer-implemented method (S20) of claim 9 or 10, characterized in that the runtime stage (S23) further comprises: managing input transmission to the PL (7) and output reception from the PL (7); configuring one or more preset split points for distributed / split inference of the inference task, enabling inference up to a layer in the DNN specified by the one or more preset split points; implementing an early exit point for scenario adaptation based on latency requirements or residual power budget of the SoC PL (2); providing a runtime status monitoring tool displaying power consumption, latency, throughput, and / or the accuracy of a current inference result.

12. The computer-implemented method (S20) of any one of claims 9 to 11, characterized in that the synthesize stage (S22) further comprises: pre-configuring the PS (3), such that the PS (3):receives an external request which relates to the inference task and is accompanied by input data, and loads configuration data of the DNN corresponding to the received inference task upon receiving the external request; and pre-configuring the PL (7), such that: an input buffer (9) of the PL (7) stores the loaded configuration data from the PS (3) under control of the PS (3), a plurality of processing elements, PEs (11), of the PL (7) performs inference on the input data under use of the DNN when the PS (3) indicates that the input buffer (9) is ready, and pre-configuring the PEs (11) in form of a systolic array.

13. The method (S20) of claim 9, characterized in that the synthesize stage (S22) enables the SoC PL (2) to be tailored to a specific industrial internet of things, IIOT, application, resulting in optimized performance and efficiency, and the runtime stage (S23) provides dynamic adaptability to changing operational conditions and task requirements in real-time.

14. A computer-implemented method (S30) of operating a System-on-Chip Programmable Logic device, SoC PL (2), i.e. the SoC PL (2) of any one of claims 1 to 8, for performing a distributed and diverse inference task, the method (S30) comprising:receiving (S31) an external request which relates to the inference task, and is accompanied by input data; loading (S32) configuration data of a deep neural network, DNN, corresponding to the received inference task upon receiving the external request; storing (S33) the loaded configuration data from a Host Processing System, PS (3) under control of the PS (3), and performing (S34), at a plurality of processing elements, PEs (11) of a Programmable Logic, PL (7), inference on the input data under use of the DNN when the PS (3) indicates that an input buffer (9) of the PL (7) is ready; characterized in that the PEs (11) form a systolic array.

15. A computer program, characterized in that the computer program comprises instructions which, when the computer program is executed by a computer, cause the computer to carry out or initiate the method (S20, S30) of any one of the claims 9 to 14 or at least one of the steps thereof.

Images