Method and electronic device for processing tensor operation workload based on open-source FPGA platform

Abstract

An electronic device for processing tensor operation workloads based on an open source FPGA platform includes a memory for storing instructions and a processor, wherein, when the instructions are executed by the processor, the electronic device receives tensor instructions from an FPGA platform based on an open source instruction set architecture, classifies and processes the received tensor instructions into tensor data operations and tensor computing operations, performs tensor data operations for data movement between external memory and internal memory, executes tensor computing operations using tensor operators in internal memory, and can control the execution of user algorithms while hiding hardware complexity through tensor virtual machine software for processing AI workloads.

Description

Technology Field

[0001] The present invention relates to the fields of artificial intelligence (AI) accelerators, semiconductor architectures, and parallel computing systems. More specifically, it relates to a Domain Specific Architecture (DSA) for efficiently processing workloads such as AI and computer vision expressed by tensor operations on an open-source Field Programmable Gate Array (FPGA) platform based on an open instruction set architecture (ISA) such as RISC-V, and to an electronic device including the same. RISC-V may refer to a Reduced Instruction Set Computer (RISC) based on an open instruction set architecture (ISA).

[0002] In addition, the present invention can be applied to fields of programming that combine a Tensor Virtual Machine (TVM) software stack that abstracts hardware complexity with a hardware architecture that combines a Very Long Instruction Word (VLIW) processor array and hardware multithreading. Background Technology

[0004] With the recent explosive growth of artificial intelligence technology, the computing performance requirements for training and inference of AI models are also increasing exponentially. These AI workloads are centered around tensor operations such as massive matrix multiplication and convolution, which can place a burden on general-purpose CPUs or GPUs in terms of computational load and memory bandwidth.

[0005] To address these issues, Domain-Specific Architectures (DSAs), which are hardware optimized for tasks in specific domains, are being actively researched, and the most widely used DSA for AI acceleration currently is the Systolic Array (SA). A Systolic Array is a structure in which data flows like a pipeline and performs parallel operations, and it can demonstrate high efficiency for regular operations such as matrix multiplication.

[0006] However, systolic array-based DSAs have the limitation of being very difficult to program. Since directly controlling the data flow and timing of an SA requires a high level of expertise, most users are forced to rely on pre-built libraries provided by hardware vendors. This can be a significant constraint in research and development environments that need to implement new AI models or custom algorithms.

[0007] Furthermore, since SAs are highly specialized for specific computations, they can be inefficient for handling various auxiliary tasks included in AI workloads, such as vision data preprocessing and data remapping. Consequently, even commercial chips (ASICs) equipped with SAs still require separate high-performance CPUs or GPUs for these tasks, which contributes to increased complexity and cost for the entire system.

[0008] Meanwhile, existing high-performance solutions based on ARM or NVIDIA may make it difficult for small and medium-sized developers or research institutions to freely develop and commercialize technology due to high licensing costs and closed ecosystems. The problem to be solved

[0010] The present invention aims to provide an electronic device of a new architecture capable of flexibly and efficiently processing AI and tensor computation workloads in an open source FPGA platform environment.

[0011] In addition, the present invention aims to overcome the low programmability and inflexibility of systolic arrays and to provide a software stack including a domain-specific language (DSL) compiler, thereby providing an environment in which software engineers can easily implement and port customized AI algorithms without a deep understanding of the hardware.

[0012] The present invention aims to maximize the overall efficiency of the system by clearly separating tensor operations into 'tensor data operations' responsible for data movement between external memory and internal memory and 'tensor computing operations' performed only in internal memory, and by providing a hardware path optimized for the characteristics of each operation.

[0013] The present invention aims to achieve a high throughput of processing one VLIW instruction per clock without memory stall cycles by combining a VLIW (Very Long Instruction Word) processor array and hardware multithreading technology to effectively hide latency occurring in the instruction pipeline.

[0014] The present invention aims to implement the overlap of computation and memory access at the hardware level by configuring two independent memory pages within each VLIW processor so that while one thread performs computation, another thread can simultaneously perform data transfer. means of solving the problem

[0016] An electronic device for processing tensor operation workloads based on an open source FPGA platform includes a memory for storing instructions and a processor, wherein, when the instructions are executed by the processor, the electronic device receives tensor instructions from an FPGA platform based on an open source instruction set architecture, classifies and processes the received tensor instructions into tensor data operations and tensor computing operations, performs tensor data operations for data movement between external memory and internal memory, executes tensor computing operations using tensor operators in internal memory, and can control the execution of user algorithms while hiding hardware complexity through tensor virtual machine software for processing AI workloads. Effects of the invention

[0018] The open source FPGA platform-based tensor computation workload processing method and electronic device according to the present invention provide a high-level software stack (DSL, TVM) that abstracts hardware complexity, thereby dramatically improving programmability and flexibility compared to existing systolic arrays, and has the effect of enabling developers to quickly implement and test customized AI algorithms.

[0019] In addition, the present invention implements prefetching-based streaming data transmission without caching through a unique architecture that separates tensor data operations and computing operations, and by ensuring that computing operations are performed only in internal memory, it has the effect of fundamentally resolving memory bottlenecks and simplifying hardware design.

[0020] The present invention can achieve a high effective execution speed of processing one VLIW instruction per clock per processor by organically combining a VLIW processor array and round-robin hardware multithreading to completely hide long instruction pipeline latency.

[0021] The present invention can increase the accessibility of AI hardware technology by enabling the implementation of high-performance AI accelerators based on a low-cost open-source RISC-V FPGA platform without high licensing costs or dependency on a specific vendor.

[0022] In addition, the present invention can reduce the overall processing time of the system by simultaneously performing memory transfer and non-linear function (e.g., activation function) processing through a 2-page internal memory structure that supports the overlap of computation and memory access and a stream processor placed in the data movement path. Brief explanation of the drawing

[0024] FIG. 1 is a drawing for explaining a system utilizing an artificial intelligence model according to one embodiment. FIG. 2 is a diagram illustrating the learning of a neural network according to one embodiment. FIG. 3 is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment. FIG. 4 is a block diagram showing the configuration of an electronic device for processing a tensor computation workload based on an open source FPGA platform according to one embodiment. FIG. 5 shows a flowchart of a method for processing a tensor computation workload based on an open source FPGA platform according to one embodiment. FIG. 6 shows a flowchart of a method for processing tensor computation workloads based on an open source FPGA platform according to one embodiment. FIG. 7 conceptually illustrates the data and program input / output flow of an electronic device according to one embodiment, and the process of executing a neural network-based tensor operation using the same. FIG. 8 is a block diagram schematically illustrating the physical architecture of a core computational accelerator (TSChip) of an electronic device according to one embodiment. FIG. 9 is a block diagram illustrating in detail the internal configuration and interface of a central tensor processor unit (dp_core) of an electronic device according to one embodiment. FIG. 10 is a block diagram illustrating in detail the internal architecture of a tensor arithmetic execution unit (core) of an electronic device according to one embodiment. FIG. 11 is a block diagram illustrating the detailed configuration of an arithmetic logic unit (ALU) included inside a VLIW processor core (pcore) of an electronic device according to one embodiment. Specific details for implementing the invention

[0025] Various embodiments are described in detail below with reference to the attached drawings. However, these embodiments may be modified in various ways, and therefore the scope of the patent application is not limited or restricted by these embodiments. All modifications and equivalent substitutes should be considered to be included within the scope of the rights. The specific structural or functional descriptions for each embodiment are merely for illustrative purposes and may be modified in various forms. Accordingly, the embodiments are not limited to a specific form, and the scope of this specification includes variations of the technical concept, equivalents, or substitutes.

[0026] The terms "first" or "second" may be used to describe various components, but are intended merely to distinguish each component. For example, the first component may be named the second component, and conversely, the second component may be named the first component. When a component is referred to as being "connected" to another component, this means that it may be directly connected or that there may be another component in between. The terms used are for descriptive purposes only and are not intended to be limiting. Singular expressions include the plural unless otherwise interpreted in the context. In this specification, expressions such as "comprising" or "having" indicate the presence of the features or elements described in the specification and should be understood as not excluding the presence of other features or elements.

[0027] Unless otherwise specifically defined, all terms used herein have the meanings generally understood by those skilled in the art. Terms defined in commonly used dictionaries should be interpreted in the context of the relevant technology and should not be interpreted in an overly formal sense unless explicitly defined in this application. Additionally, when describing the drawings, the same reference numerals are used for identical components, and redundant descriptions are omitted. When describing embodiments, if a detailed description of the relevant technology would become unnecessarily obscured, such detailed description may be omitted.

[0028] The embodiments can be implemented in various forms, such as personal computers, laptops, tablets, smartphones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. An artificial intelligence (AI) system is a computer system that implements human-level intelligence; unlike existing rule-based smart systems, it is a system in which the machine learns and makes decisions on its own. As the use of AI systems increases, recognition rates improve and user preferences are understood more accurately, leading to the gradual replacement of existing systems with deep learning-based AI.

[0029] Artificial intelligence technology consists of machine learning and various component technologies that utilize it. Machine learning refers to algorithms that autonomously classify and learn the characteristics of input data, while component technologies are techniques that mimic human cognitive and judgment functions through machine learning algorithms, such as deep learning. These technologies include linguistic understanding, visual understanding, reasoning and prediction, knowledge representation, and motion control.

[0030] The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding refers to the technology of recognizing and processing human language, including natural language processing, machine translation, dialogue systems, question answering, and speech recognition and synthesis. Visual understanding refers to the technology of recognizing and processing objects, including object recognition, tracking, image search, human recognition, and scene understanding. Inference and prediction refer to the technology of judging and predicting information, including knowledge-based inference, optimization prediction, and recommendation systems. Knowledge representation refers to the technology of automatically processing information from human experience, including knowledge construction and management. Motion control refers to the technology of controlling the movement of autonomous vehicles or robots, including navigation, collision avoidance, and driving control. Generally, machine learning algorithms are trained using a trial-and-error method to apply them to real-life situations. Deep learning, in particular, requires hundreds of thousands of iterations. When it is difficult to execute this in a real-world environment, training is conducted through simulations by implementing a virtual environment on a computer.

[0031] In this invention, Artificial Intelligence (AI) refers to a technology that implements human learning, reasoning, and perceptual abilities on a computer, and includes concepts such as machine learning and symbolic logic. Machine learning (ML) is an algorithmic technology that learns the characteristics of input data on its own. Through such machine learning algorithms, AI technology can analyze input data, learn the results, and perform judgments or predictions based on them. Furthermore, technologies that mimic human cognitive and judgment functions using machine learning algorithms can also be included in the category of AI. For example, fields such as linguistic understanding, visual understanding, reasoning and prediction, knowledge representation, and motion control fall into this category. Machine learning refers to the process of training a neural network model through experience in processing data. Through this, computer software can improve its data processing capabilities on its own. A neural network model models the correlations between data and can be expressed by various parameters. The core of machine learning is to optimize the model's parameters by extracting and analyzing features from given data to derive relationships between the data and repeating this process. For example, a neural network model can learn the relationship between input and output, or learn relationships by deriving regularities solely from the input data.

[0032] Artificial intelligence learning models or neural network models are designed to implement the structure of the human brain on a computer and include multiple network nodes that mimic neurons. These nodes are interconnected to exchange signals and process data through layers of varying depths within the AI ​​learning model. Such models may include artificial neural networks and convolutional neural networks (CNNs). For example, AI learning models can be machine learned through methods such as supervised learning, unsupervised learning, and reinforcement learning. Machine learning algorithms that may be used include decision trees, Bayesian networks, support vector machines, artificial neural networks, AdaBoost, perceptrons, genetic programming, and clustering.

[0033] CNNs are a type of multi-layer perceptron designed with minimal preprocessing, consisting of one or more convolutional layers and general artificial neural network layers. Thanks to this structure, CNNs can effectively utilize 2D input data and demonstrate excellent performance in image and speech fields. CNNs are trained via standard backpropagation and have the advantage of being easier to train and using fewer parameters than other feedforward artificial neural network techniques. Convolutional networks are neural networks containing a set of nodes with bound parameters, and many computer vision tasks have been improved as the amount of training data increases and computational power improves. In today's large-scale datasets, overfitting is not critical, and increasing the network size improves test accuracy. Since the optimal use of computing resources becomes a limiting factor, distributed and scalable implementations of deep neural networks become necessary.

[0035] FIG. 1 is a diagram illustrating a system utilizing an artificial intelligence model according to one embodiment.

[0036] As illustrated in FIG. 1, a system (100) utilizing an artificial intelligence model may include multiple user terminals (110-1, ���), a server (120), and a database (130). According to one embodiment, the database (130) is configured separately from the server (120), but it is not necessary to configure it this way, and the database (130) may be located inside the server (120). For example, the server (120) may include multiple artificial intelligences for performing machine learning algorithms. According to another embodiment, multiple user terminals (110-1, ���), a server (120), and a database (130) may be connected to communicate with each other through a network (N).

[0037] The network (N) can perform wireless or wired communication between multiple user terminals (110-1, ���), a server (120), and a database (130). For example, the network can support wireless communication in the form of LTE (long-term evolution), LTE-A (LTE Advanced), CDMA (code division multiple access), WCDMA (wideband CDMA), WiBro (Wireless BroadBand), WiFi (wireless fidelity), Bluetooth, NFC (near field communication), GPS (Global Positioning System), or GNSS (global navigation satellite system). Additionally, the network (N) can perform wired communication in the form of USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard 232), or POTS (plain old telephone service).

[0038] The database (130) has the ability to store various data. The data stored in the database (130) is data acquired, processed, or used by at least one component of a multiple user terminal (110-1, ���) or server (120), and may include software (e.g., programs). The database (130) may include volatile and / or non-volatile memory.

[0040] FIG. 2 is a diagram illustrating the learning of a neural network according to one embodiment.

[0041] As shown in FIG. 2, the learning device can train a neural network (123) to process review responses received from multiple user terminals (110-1, ���) item by item. Additionally, the learning device can train a neural network (123) to extract user stay records based on user movement path information. According to one embodiment, the learning device may be a separate entity from the server (120), but this is not limited to this.

[0042] The neural network (123) includes an input layer (121) and an output layer (125), and training samples are input to generate training outputs, which are learned based on the difference between these outputs and labels. Labels may be defined from items related to review responses and user dwell records based on movement path information, and the neural network (123) is composed of several groups of nodes defined by weights and activation functions between the nodes.

[0043] The learning device can train a neural network (123) using Gradient Descent (GD) or Stochastic Gradient Descent (SGD) techniques. GD is a method of adjusting the model's parameters using the entire dataset, while SGD is a method that enables faster learning by using only a randomly selected portion of data. The learning device can calculate the training error through a loss function designed in the form of Mean Square Error (MSE) or entropy, and optimize the weights that affect the training error using backpropagation techniques.

[0044] According to one embodiment, the learning device extracts a first object from a review response to obtain a first label and performs learning with a first training output generated by applying it to a first neural network. Additionally, it extracts a second object from movement path information to obtain a second label and performs learning with a second training output generated by applying it to a second neural network.

[0045] The learning device can generate a first training feature vector based on the composition, location, and pattern features of review responses, and a second training feature vector based on the composition, length, and pattern features of movement path information. These feature vectors are applied to each neural network to generate training outputs, which are then used, along with the corresponding labels, to train the review item extraction algorithm and the user dwell record acquisition algorithm. In this context, backpropagation refers to a learning method that adjusts the weights of each layer while propagating errors generated in the output layer toward the input layer.

[0047] FIG. 3 is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment.

[0048] According to one embodiment, an artificial intelligence model may be composed of an input layer, a hidden layer, and an output layer.

[0049] The input layer is the layer associated with the values ​​input into the artificial intelligence model. In the hidden layer, feature maps can be generated by performing multiply-accumulate (MAC) operations and activation operations on the input values. The MAC operation is a process of multiplying the input values ​​by their corresponding weights and then summing these multiplied values. The activation operation is a process of inputting the results of the MAC operation into an activation function to output a final result, and various types of activation functions can be used. For example, activation functions may include, but are not limited to, the sigmoid function, tangent function, ReLU function, Leaky ReLU function, Max Out function, and ELU function.

[0050] A hidden layer may consist of at least one layer, for example, divided into a first hidden layer and a second hidden layer. In this case, the first hidden layer generates a feature map by performing MAC operations and activation operations based on the input values ​​of the input layer, and this feature map can be used as the input value of the second hidden layer. The second hidden layer can perform MAC operations and activation operations based on the feature map generated as a result of the first hidden layer.

[0051] The output layer may be a layer related to the result of an operation performed in the hidden layer.

[0052] In one embodiment, a learning model learns syllable (character) patterns that are frequently used together in a given corpus to automatically identify the boundaries between compound words and named entities, and integrates object information from a first UI source with object information rendered in a browser to generate a learning object information file. Using this learning object information file, data for training a deep learning network is generated, and based on data received from various domains, the data is standardized into an integrated format according to one or more standardization methods suitable for each domain. Subsequently, data of a specific domain is trained and inferred, information necessary for standardization in that domain is determined, and post-processing can be performed on the data received from various domains.

[0053] The first UI source includes an XML file, and the training object information file includes an input JSON file for feature learning and an output JSON file used as label data during training. This output JSON file is a file containing HTML DOM Tree information implemented in compliance with web standards. Various domains include at least one of a RAN (radio access network), a transport, or a core, and the post-processing process may include a correlation function.

[0055] FIG. 4 is a block diagram showing the configuration of an electronic device for processing a tensor computation workload based on an open source FPGA platform according to one embodiment.

[0056] An electronic device (400) according to one embodiment may include a processor (420) and a memory (430), and some of the illustrated components may be omitted or substituted. An electronic device (400) according to one embodiment may be a server or a terminal. According to one embodiment, the processor (420) is a component capable of performing operations or data processing regarding the control and / or communication of each component of the electronic device (400), and may be composed of one or more processors. The memory (430) may store information related to the method described above or store a program in which the method described above is implemented. The memory (430) may be volatile memory or non-volatile memory. The memory (430) may store various file data, and the stored file data may be updated according to the operation of the processor (120).

[0057] According to one embodiment, the processor (420) can execute a program and control the device (400). The code of the program executed by the processor (120) can be stored in memory (430). Operations of the processor (420) can be performed by loading instructions stored in memory (430). The electronic device (400) can be connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown in the drawing) and exchange data.

[0058] An electronic device (400) according to one embodiment includes a processor (420) and a memory (430). An electronic device (400) according to one embodiment may be the server or terminal described above. The processor (420) may include at least one device described through FIGS. 1 to 3 or perform at least one method described through FIGS. 1 to 3. The memory (430) may store information related to the method described above or store a program in which the method described above is implemented. The memory (430) may be volatile memory or non-volatile memory.

[0061] FIG. 5 shows a flowchart of a method for processing a tensor computation workload based on an open source FPGA platform according to one embodiment.

[0062] Although process steps, method steps, algorithms, etc. are described in a sequential order in the flowchart of FIG. 5, such processes, methods, and algorithms may be configured to operate in any suitable order. In other words, the steps of the processes, methods, and algorithms described in various embodiments of the present invention do not need to be performed in the order described in the present invention. Furthermore, even if some steps are described as being performed asynchronously, in other embodiments, such steps may be performed simultaneously. Also, the example of a process by the depiction in the drawings does not mean that the illustrated process excludes other variations and modifications therefrom, does not mean that any of the illustrated process or its steps is essential to one or more of the various embodiments of the present invention, and does not mean that the illustrated process is preferred.

[0063] In operation 510, the electronic device (400) can receive tensor instructions from an open-source FPGA platform and classify them into data and computing operations. This may include receiving tensor operation instructions from a host processor that follows an open instruction set architecture (ISA), such as RISC-V, as a starting point for AI workload processing. As an example of implementation, a DP core (dp_core), which is a central control unit within the electronic device (400), can fetch tensor instructions from the host via a standard bus protocol such as AXILite. The electronic device (400) can analyze the nature of the instructions and clearly classify them into two independent types: Tensor Data Operations and Tensor Computing Operations. This selective classification serves as a preliminary step to assign subsequent operations to hardware paths optimized for their respective characteristics, thereby maximizing the computational efficiency of the entire system.

[0064] In operation 520, the electronic device (400) can move data between external and internal memory through tensor data operations and execute computing operations internally. Tensor data operations can handle the transfer of data between main memory, such as external DDR memory, and high-speed internal memory (such as register banks) located within the VLIW processor array. This process can be efficiently carried out without CPU intervention through Direct Memory Access (DMA) transfer processing blocks (ddr_rx, ddr_tx), and round-trip latency can be concealed by supplying data in a streaming manner through prefetching. On the other hand, tensor computing operations refer to actual arithmetic logic operations such as matrix multiplication and convolution, which can be performed only on data loaded into internal memory without accessing external memory. By separating the types of operations and limiting the execution area in this way, there is an effect of fundamentally eliminating memory access latency or stall cycles that may occur during computing operations. This allows the hardware's computational throughput to be maintained at its maximum without a complex cache hierarchy, thereby simplifying hardware design and maximizing performance.

[0065] In operation 530, the electronic device (400) can hide hardware complexity and execute user algorithms through Tensor Virtual Machine (TVM) software. This may mean providing an abstraction layer between the hardware and the user application. Using a domain-specific language (DSL), the user can write high-level tensor operation code such as 'z[i] = x[i] + y[i]' without knowing the detailed structure of the hardware (VLIW instructions, pipelines, memory structures, etc.). Then, a compiler included in the TVM can analyze this code and automatically convert it into optimal low-level VLIW instruction code and data movement schedules that the hardware can execute. Unlike existing methods that required a deep understanding of the hardware, such as systolic arrays, this approach has the effect of dramatically improving the development productivity of software engineers. Additionally, the same user code can be ported to various hardware configurations with different numbers of processor cores or memory sizes simply by recompiling it.

[0066] An electronic device (400) according to one embodiment receives tensor instructions on an FPGA platform based on an open source instruction set architecture, classifies and processes the received tensor instructions into tensor data operations and tensor computing operations, performs tensor data operations for data movement between external memory and internal memory, executes tensor computing operations using tensor operators in internal memory, and can control to hide hardware complexity and execute user algorithms through tensor virtual machine software for AI workload processing.

[0068] FIG. 6 shows a flowchart of a method for processing tensor computation workloads based on an open source FPGA platform according to one embodiment.

[0069] Although process steps, method steps, algorithms, etc. are described in a sequential order in the flowchart of FIG. 6, such processes, methods, and algorithms may be configured to operate in any suitable order. In other words, the steps of the processes, methods, and algorithms described in various embodiments of the present invention do not need to be performed in the order described in the present invention. Furthermore, even if some steps are described as being performed asynchronously, in other embodiments, such steps may be performed simultaneously. Also, the example of a process by the depiction in the drawings does not mean that the exampled process excludes other variations and modifications therefrom, does not mean that any of the exampled process or its steps is essential to one or more of the various embodiments of the present invention, and does not mean that the exampled process is preferred.

[0070] In operation 610, the electronic device (400) can have a DP core receive tensor instructions from a RISC-V host and secure 16 hardware thread spaces. This step is a preparatory process for tensor instruction execution and can begin with the fetch unit (dp_fetch) of the DP core, which is the central control unit, receiving instructions from the RISC-V host. The received instructions are not simply executed sequentially, but can be connected to hardware threads for parallel processing. In particular, there may be 16 hardware execution threads within the VLIW processor (PCore) to hide instruction pipeline latency, and the DP core can decode instructions to allocate register space for each thread to use and prepare a context for execution. This is a process of securing hardware resources in advance for fine-grained parallelism, which can have the effect of laying the foundation for performing computations with high throughput without latency in subsequent steps. The number of hardware execution threads is merely an example and may vary depending on the configuration.

[0071] In operation 620, the electronic device (400) can read data from external DDR memory, stream process it, and store it in internal memory (register bank). This may correspond to a data staging stage that prepares operands required for tensor computing operations. Under the control of the DP core, the DDR-RX block can read data from external DDR memory via DMA. At this time, the data is not stored directly in internal memory but may pass through a stream processor placed along the data path. The stream processor can apply non-linear transformations in real time, such as data mapping, dimension scaling, or activation functions (e.g., ReLU) for AI workloads, just before the data is written to memory. By processing operations inline during the data movement process in this way, the process of processing activation functions by consuming a separate operation cycle after storing the data can be omitted. This can shorten the overall execution cycle of the system and reduce memory bandwidth usage.

[0072] In operation 630, the electronic device (400) can have multiple PCore arrays that can execute VLIW, 8-ALU, FMA operations in parallel on internal memory data. This may be the core operation stage where actual tensor computing takes place. Multiple VLIW processor cores (PCores) that receive execution commands from the DP core can perform parallel operations simultaneously using data stored in internal memory (register bank). Each PCore may have an architecture that controls multiple operations with a single instruction (VLIW) and may include 8 Arithmetic Logic Units (ALUs) for simultaneous operations on 8 elements of a vector word. In particular, to accelerate AI workloads where operations of the form (aXb)+c are frequent, Fusion Multiplication-Accumulation (FMA) operations can be processed in a single clock cycle in combination with an X register file that stores int32 accumulator values. This multi-layered parallel processing structure has the effect of providing overwhelming computational throughput for tensor operations. The number of ALUs is just an example and may vary depending on the settings.

[0073] In operation 640, the electronic device (400) can transfer the operation result to external memory and convert high-level code into VLIW through a DSL compiler. When the tensor computing operation is completed, the result value stored in the internal memory of the PCore can be transferred back to external DDR memory via a DDR-TX block or retained in internal memory for subsequent operations. All these processes, namely the generation of data movement instructions, the generation of VLIW operation instructions, and the analysis of dependencies and scheduling between them, can be performed automatically by a Domain Specific Language (DSL) compiler. The compiler performs the role of converting the user's relatively complex code into simple code that maximizes the hardware's parallel processing capabilities. As a result, it can achieve the effect of shortening the development time of high-performance AI applications and facilitating maintenance by providing an easy programming environment for developers while maximizing the potential performance of the hardware.

[0074] An electronic device (400) according to one embodiment receives tensor instructions from a RISC-V host via an AXILite bus through a DP core, which is a central control unit of an FPGA platform based on a RISC-V instruction set architecture, divides the tensor instructions received from the DP core into 16 hardware threads to create registers for each thread and secure an instruction execution space, and when performing tensor data operations, reads data from external DDR memory via DDR-RX, performs data mapping in a stream processor, and stores it in a register bank which is internal memory, and when performing tensor computing operations, executes parallel operations on data stored in internal memory by utilizing a plurality of PCores configured as a VLIW (Very Long Instruction Word) processor array, wherein each PCore includes 8 ALUs (Arithmetic Logic Units) to simultaneously perform arithmetic operations on 8 elements of a vector word, and stores an int32 accumulator value in an X register file within the PCore to perform a fused multiply-add (FMA) operation in the form of (aXb)+c. It processes operations in a single cycle, stores integer values ​​for loop counters, array indices, and pointer references in the I register file, performs integer operations via IALU, supports nested execution where the register bank consists of two internal memory pages so that each page is associated with one of two hardware threads, allowing another thread to perform memory transfers while one thread performs operations, stores temporary data during tensor data transfer in the SRAM core acting as scratch memory, and controls the transfer of tensor operation results to external memory via DDR-TX or their retention in internal memory for the next operation,Through a Domain-Specific Language (DSL) compiler, high-level tensor operation code written by the user is converted into hardware-executable VLIW instructions, allowing control that hides hardware details while enabling the same software to be ported to hardware of various performance levels simply by recompiling.

[0076] An electronic device (400) according to one embodiment monitors the execution pattern of tensor operations at runtime to identify frequently used operation sequences and dynamically compiles them into fused instructions and stores them in an instruction cache; to accelerate sparse tensor operations, it stores the indices and values ​​of non-zero elements in a compressed form and performs conditional execution by utilizing the VMASK register of the VLIW processor to skip operations on zero elements; to optimize energy efficiency, it analyzes the precision requirements of tensor operations to apply approximate computing to small values ​​below a threshold and verifies in real time whether the degradation of the operation result quality is within an acceptable range; to support a multi-tenant environment, it assigns a context ID to each tensor instruction and schedules instructions of different contexts to share hardware resources in a time-sharing manner, while hardware-verifying access rights per memory page to ensure data isolation between contexts; and to support debugging and profiling, it measures the execution cycles, memory access patterns, and pipeline stall frequency of each VLIW processor using a hardware counter and stores this in a trace buffer. It is possible to record to identify performance bottlenecks in real time, and to provide a mechanism for fault tolerance by redundantly executing critical operations on two or more VLIW processors and comparing the results to re-execute in case of a discrepancy or select the correct result by majority voting.

[0077] An electronic device (400) according to one embodiment may include various advanced control functions for runtime optimization, sparse tensor acceleration, energy efficiency optimization, multi-user support, debugging, and ensuring stability.

[0078] According to one embodiment, an electronic device (400) can monitor the execution pattern of tensor operations at runtime to identify frequently used operation sequences and dynamically compile them into fused instructions and store them in an instruction cache. This can be implemented by a hardware-level monitoring unit detecting a pattern in which convolution operations and activation function (ReLU) operations appear repeatedly in succession. The detected sequences can be combined into a single VLIW instruction through a microcompiler embedded on the chip. The fused instructions are stored in a special instruction cache and can be used immediately upon subsequent calls. This dynamic fusion can reduce the cycles required for instruction fetching and decoding, thereby having the effect of further accelerating the execution performance of a specific AI model.

[0079] Additionally, the electronic device (400) can store the indices and values ​​of non-zero elements in a compressed form to accelerate sparse tensor operations and perform conditional execution by utilizing the VMASK register of the VLIW processor to skip operations on zero elements. Tensors used in natural language processing or recommendation systems may have sparse characteristics in which most elements are zero. To process such tensors, the electronic device (400) can store only non-zero values ​​and their location information in memory in a compressed format (e.g., CSR, COO). During computation, the VMASK register can be dynamically set based on this location information to disable the ALU operation of the vector lane corresponding to zero. This allows unnecessary operations, such as multiplication with zero, to be skipped at the hardware level, thereby enabling significant reduction in computation time and power consumption when processing sparse tensors.

[0080] According to one embodiment, the electronic device (400) can analyze the precision requirements of tensor operations for energy efficiency optimization, apply approximate computing to small values ​​below a threshold, and verify in real time whether the degradation of the quality of the computation result is within an acceptable range. This can be implemented by using fewer bits or simplified computation logic instead of performing computation with full precision for small values ​​that have a negligible impact on the final accuracy of the AI ​​model. For example, when an operand smaller than a specific threshold is input, the ALU may operate in a low-power approximate computing mode. At the same time, a monitoring unit at the output stage can track the cumulative error or statistical distribution of the computation result to control the accuracy of the entire AI model so that it does not exceed an acceptable range. Such approximate computing techniques can be effective in minimizing power consumption, particularly in battery-operated edge devices.

[0081] According to one embodiment, the electronic device (400) may assign a context ID to each tensor instruction to support a multi-tenant environment and schedule instructions of different contexts to share hardware resources in a time-sharing manner. Additionally, to ensure data isolation between contexts, access rights per memory page may be verified hardware-wise. This may be a function designed for situations where multiple users share a single accelerator, such as in a cloud computing environment. Each user's task is assigned a unique context ID, and the scheduler of the DP core, which is the central control unit, can execute instructions of different tasks in a fair time-sharing manner based on this ID. At this time, the hardware memory management unit (MMU) verifies the internal memory page addresses accessible by each context ID, thereby fundamentally blocking one task from accessing the data of another task. Through this, strong security and data isolation can be ensured while increasing the utilization rate of hardware resources.

[0082] According to one embodiment, the electronic device (400) can measure the execution cycles, memory access patterns, and pipeline stall frequency of each VLIW processor using hardware counters to support debugging and profiling, and record this in a trace buffer to enable real-time identification of performance bottlenecks. Dedicated hardware counters for performance analysis may be embedded within each PCore. These counters can calculate the frequency of specific events (e.g., number of executed instructions, number of memory reads / writes, pipeline delay cycles) with minimal impact on program execution. The measured data is sequentially stored in an on-chip trace buffer, and the developer can read the contents of this buffer to precisely analyze where in the software performance degradation occurs. This has the effect of providing essential information for the developer to optimize code to fully utilize the characteristics of the hardware architecture.

[0083] According to one embodiment, the electronic device (400) can be controlled to provide a mechanism for fault tolerance in which critical operations are executed redundantly on two or more VLIW processors and the results are compared to re-execute the operation if a discrepancy occurs, or to select the correct result by majority voting. This may be a function for fields requiring high reliability, such as automobiles and aerospace. When a specific operation is designated as a 'critical operation,' the scheduler can assign the operation to two or more PCores simultaneously to perform the same operation redundantly. The result values ​​from each PCore are compared in real time through a hardware comparator, and if the values ​​do not match, it is determined to be a temporary hardware error and the operation is re-executed, or if executed on three or more cores, the correct value is adopted through majority voting. Such redundancy and voting mechanisms can significantly improve the reliability and stability of the system.

[0084] The fusion instructions, sparse tensor processing methods, thresholds for approximation operations, multi-tenant scheduling policies, profiling target events, and fault recovery methods described herein are merely examples and are not limited thereto; they may vary depending on the application or system configuration.

[0086] FIG. 7 conceptually illustrates the data and program input / output flow of an electronic device (400) according to one embodiment, and the process of executing a neural network-based tensor operation using the same.

[0087] Referring to FIG. 7, the electronic device (400) can perform a complex tensor operation workload based on 'input data' to be processed and an 'input program' that defines the processing method and generate 'output data'.

[0088] According to one embodiment, the electronic device (400) may receive input data and an input program from an external source to execute tensor operations. For example, the input data may be a multidimensional array (tensor) predefined for a specific task. As illustrated in FIG. 7, when performing an image classification task such as MNIST handwritten digit recognition, the input data may be a vector having 784 numeric values ​​(e.g., pixel brightness values ​​between 0 and 255) that are one-dimensionally unfolded from a 28x28 pixel grayscale image. The input program may be a neural network algorithm program precompiled to run on a RISC-V FPGA platform and may internally consist of a sequence of tensor data operations and tensor computing operation instructions.

[0089] According to one embodiment, the electronic device (400) can sequentially execute tensor operations according to the structure of the neural network model defined in the input program. When the input program is loaded, the DP core, which is a central control unit, can interpret the instructions of the program and coordinate the entire inference process. For example, the process of transmitting signals from 784 nodes of the input layer to 128 nodes of the hidden layer 1 can be implemented as a matrix-vector multiplication operation between a weight matrix of size 128x784 and an input vector of size 784x1. For this operation, the DP core can first instruct a tensor data operation to load the weight matrix and the input data vector stored in external DDR memory into internal memory (e.g., the register bank of each VLIW processor).

[0090] According to one embodiment, the electronic device (400) can perform large-scale parallel computing by utilizing a VLIW processor array after data loading into internal memory is completed. The DP core transmits tensor computing operation commands to the VLIW processor (PCore) array, and each PCore can execute a portion of matrix-vector multiplication in parallel for the allocated data. At this time, computational efficiency can be maximized by processing multiplication and addition in a single cycle by utilizing hardware acceleration functions such as FMA (Fused Multiply-Add) operations. After the matrix-vector multiplication is completed, the process of applying an activation function (e.g., relu) of hidden layer 1 can be processed in-line by a stream processor located in the path that transmits the operation result externally, or the VLIW processor can directly perform it as a subsequent operation.

[0091] According to one embodiment, the electronic device (400) can complete the entire neural network operation by using the result value as the input to the next layer when one layer operation is completed. The output of Hidden Layer 1 (128 node values) becomes the input to Hidden Layer 2, undergoing matrix-vector multiplication with another weight matrix (e.g., 64x128) and the application of the relu activation function. This process is repeated up to the Output Layer, and finally, at the Output Layer, a softmax function is applied to 10 node values ​​to generate output data containing probability values ​​for each class (e.g., numbers 0 through 9). This final result can then be transferred to external memory under the direction of the DP core and utilized in another system.

[0092] The structure of the neural network described herein (e.g., number of hidden layers, number of nodes, type of activation function) and the form of the input data (e.g., image vector) are merely examples of the present invention and are not limited thereto, and can be configured and executed as any tensor operation graph depending on the content of the input program.

[0094] An electronic device (400) according to one embodiment independently configures an X register file for storing int32 accumulator values ​​for fused multiplication-accumulation (FMA) operations and an I register file for storing integer values ​​for loop counters, array indices, and pointer operations within each VLIW processor that performs tensor computing operations, and has an integer-only IALU (Integer ALU) separate from the ALU array for vector operations to separate vector data processing and control flow operations at the hardware level, supports conditional execution by individually enabling / disabling each lane of the vector through the VMASK register to perform data-dependent operations without branch instructions, places a stream processor in the data transfer path between external memory and internal memory to apply up to four different data mapping functions in real time immediately before data is written to memory or immediately after data is read, and processes the activation function of the AI ​​workload inline in the stream processor to control the completion of non-linear transformation simultaneously with memory transfer without a separate activation function processing cycle.

[0095] According to one embodiment, the electronic device (400) can optimize the data processing path by independently configuring dedicated register files and operation units according to the type of operation within each VLIW processor (PCore) that performs tensor computing operations. Specifically, a 32-bit X register file for storing intermediate results (accumulated values) of complex vector operations such as fused multiplication-accumulation (FMA) and an I register file for storing integer values ​​used in the control flow of a program, such as loop counters, memory address calculations, and array indexing, can be provided separately. Correspondingly, an IALU (Integer ALU) that performs only integer operations for control flow can be additionally placed separately from the ALU array dedicated to arithmetic operations of vector data. By separating the vector data path and the integer-based control / address path at the hardware level in this way, data operations and address calculations can be performed independently in parallel within a single VLIW instruction without contention for each other's resources. This can have the effect of maximizing pipeline efficiency and increasing computational throughput.

[0096] According to one embodiment, the electronic device (400) can support conditional execution by individually enabling or disabling each lane of the vector through a VMASK register. This may be a technique to prevent performance degradation, such as pipeline flushing, which can occur due to traditional branch instructions. VMASK is a bitmask register, where each bit corresponds one-to-one with individual ALU lanes that perform parallel operations within the vector processor. By setting the bit value of the VMASK register according to a specific condition, the final result may not be written to memory or a register even if an operation is performed in a lane where the corresponding bit is disabled (e.g., 0). Through this method, while all lanes execute the same instruction stream (SIMD, Single Instruction Multiple Data), the result reflection can be selectively controlled based on the data value, making it possible to efficiently process data-dependent operations, such as if-then-else, without branching.

[0097] According to one embodiment, the electronic device (400) may place a stream processor in the data transfer path between external memory and internal memory to perform data transformation in real time simultaneously with data transfer. This stream processor may be a hardware block that passes through immediately before data is written to the internal memory of the VLIW processor, or immediately after data is read from the internal memory. Internally, it stores up to four or more different data mapping functions (e.g., tensor transpose, data rearrangement, non-linear transformation, etc.) and can apply a specified function to the data stream in real time according to an instruction. In particular, activation functions (e.g., ReLU), which are frequently used in AI workloads, can be processed in-line in this stream processor. This eliminates the process of calculating the activation function by consuming a separate operation cycle after data loading, thereby having the effect of completing non-linear transformations by utilizing the time associated with memory transfer. This can reduce overall computational latency and increase system efficiency.

[0098] The types of X / I registers, the number of bits in VMASK registers, and the number of functions supported by the stream processor described herein are merely examples and are not limited thereto; they can be designed and configured in various ways depending on the target performance or application field.

[0100] FIG. 8 is a block diagram schematically illustrating the physical architecture of a core computational accelerator (TSChip) of an electronic device (400) according to one embodiment.

[0101] Referring to FIG. 8, the electronic device (400) may include a RISC-V host processor and a tensor operation accelerator (TSChip) that interacts with external memory (DMA). The TSChip of the electronic device (400) may be connected to an external system through two main interfaces. First, the axilite_* interface is a control bus connected to the RISC-V host and may be a low-bandwidth communication path through which the host transmits tensor operation commands to the TSChip. Second, the axi_* interface is a high-bandwidth data bus for exchanging data with external memory (e.g., DDR SDRAM) and may perform the role of efficiently transmitting a large amount of tensor data through the Direct Memory Access (DMA) method.

[0102] According to one embodiment, the electronic device (400) may include a dp_core, which is a central tensor processor unit that coordinates all activities within the TSChip, as a core control unit. The dp_core may receive tensor instructions from a RISC-V host via an axilite bridge, interpret them, and convert them into necessary sub-operations. Specifically, the dp_core may control the operation of ddr_rx and ddr_tx, which are DMA transfer blocks for tensor data operations, to direct data input / output with external memory. Additionally, it may manage all data flow and operation scheduling within the TSChip, such as commanding the core unit, which is an actual arithmetic execution unit, to start operations for tensor computing operations, and storing or reading temporary data required for operations in the sram_core.

[0103] According to one embodiment, the electronic device (400) may be equipped with hardware blocks dedicated to data movement to separate and parallelize computation and data transmission. ddr_rx is a DMA transfer processing block that receives tensor data from external DDR memory and can bring a large amount of input data into the core's internal memory via the AXI bus under the direction of dp_core. Conversely, ddr_tx can serve as a DMA transfer processing block that transmits the computation results from the core's internal memory to external DDR memory. sram_core is a high-speed on-chip scratch memory block that can be used to store temporary data generated during tensor data transmission or intermediate results reused across multiple computation steps. This can have the effect of improving the performance of the entire system by reducing repetitive access to external memory.

[0104] According to one embodiment, the electronic device (400) may include a core unit as an execution unit that performs actual tensor arithmetic operations. The core unit may be a tensor arithmetic execution unit composed of a plurality of lightweight VLIW (Very Long Instruction Word) processors arranged in an array, rather than a single processor. This core unit receives specific operation execution instructions (e.g., matrix multiplication, convolution, etc.) from dp_core and can perform highly parallelized operations using data already loaded in its internal memory. This separation of roles between the core that performs actual operations and the dp_core that is responsible for data movement and control allows each unit to focus only on its own tasks, thereby reducing hardware complexity and maximizing overall processing efficiency.

[0105] The names of each component described herein (e.g., dp_core, sram_core), types of bus protocols (e.g., AXILite, AXI), etc. are merely examples of the present invention and are not limited thereto; they may be replaced with other names or interfaces performing equivalent functions depending on the design goals or implementation environment of the system.

[0107] An electronic device (400) according to one embodiment divides the internal memory of each VLIW processor into two independent physical pages, configuring the first page to be simultaneously usable for executing tensor operations of a first hardware thread and the second page for DMA data transfer of a second hardware thread, and parallelizes data transfer between different memory layers by simultaneously executing up to two tensor data operations in a DP core, which is a central control unit, while independently performing transfer between the core internal memory and external DDR and transfer between the scratchpad memory and the core internal memory, and prevents memory access conflicts by placing private data and shared data within each memory page in different address spaces so that the private memory words of each of the 16 threads are stored in a thread-interleaving manner and data shared by all threads is stored in a contiguous address space, and dynamically rearranges the execution order of tensor instructions to execute instructions without data dependencies out of order, while guaranteeing sequential execution only when the application explicitly enforces the order, and eliminates round-trip latency by transmitting external memory data in a streaming manner through prefetching so that tensor data operations required for subsequent execution By precisely specifying data, you can control the maximum utilization of memory bandwidth even without caching.

[0108] According to one embodiment, the electronic device (400) can maximize the overlap of computation and data transfer by dividing the internal memory of each VLIW processor into two independent physical pages. Specifically, while the VLIW processor executes a tensor computing operation using data from a first memory page allocated to a first hardware thread, the DMA controller can simultaneously load data to be used next by a second hardware thread from external memory into a second memory page. When the operation is completed, the roles of the two pages are ping-ponged, allowing the computation unit to start the next operation without interruption. This structure has the effect of completely hiding the time required for slow external memory access behind the computation time, thereby minimizing the idle time of the computation unit and maximizing the throughput of the entire system.

[0109] According to one embodiment, the electronic device (400) can increase the parallelism of data movement by simultaneously executing up to two or more tensor data operations in a DP core, which is a central control unit. This may mean processing data transfer between different memory layers in parallel independently. For example, data transfer between the core's internal memory and external DDR memory may occur simultaneously, while data transfer between the scratch-pad temporary memory and the core's internal memory may be performed independently. Additionally, to increase the data placement efficiency of internal memory pages, private data used independently by each of the 16 execution threads and shared data shared by all threads can be stored separately. Private data can be stored in an interleaved manner where memory words of each thread appear alternately to reduce contention during simultaneous access, and shared data can be stored in a contiguous address space to increase access efficiency.

[0110] According to one embodiment, the electronic device (400) can support out-of-order execution by analyzing the data dependencies of tensor instructions and dynamically rearranging the execution order. The scheduler of the DP core can identify instructions that do not have data dependencies on each other and execute them first regardless of the program order in the order in which resources are available, thereby increasing the efficiency of the pipeline. However, if the application explicitly sets a synchronization barrier, sequential execution can be guaranteed to maintain the accuracy of the program. In addition, the architecture of the present invention can achieve high memory bandwidth utilization without complex cache hardware. This is because the tensor data operation instruction accurately specifies the data required for subsequent computing operations, so the compiler or runtime scheduler uses a streaming method to prefetch the corresponding data. By utilizing such predictable data access patterns, it has the effect of providing deterministic high performance without performance degradation due to cache misses.

[0111] The number of pages in internal memory, the number of concurrently executable data operations, the data storage method, and the out-of-order execution policy described herein are merely examples and are not limited thereto; they can be configured in various ways depending on the size of the chip or the characteristics of the target application.

[0113] FIG. 9 is a block diagram illustrating in detail the internal configuration and interface of a central tensor processor device (dp_core) of an electronic device (400) according to one embodiment.

[0114] Referring to FIG. 9, the dp_core of the electronic device (400) serves as a control hub for tensor operation workloads and may include various subcomponents responsible for receiving, interpreting, and distributing instructions. The dp_core is externally connected to an axilite bridge, core, sram_core, ddr_rx, ddr_tx, etc., via a number of buses (bus_*, task_*, readmaster*, writemaster*), and can manage all data flow and operation execution within the TSChip.

[0115] According to one embodiment, the electronic device (400) receives tensor instructions from a RISC-V host through a dp_fetch unit and can branch a processing path depending on the type of instruction. dp_fetch decodes the instruction transmitted through the bus_* interface, and if the instruction is a 'tensor operator execution' instruction that directs actual arithmetic operations, it can be directly transmitted to the core, which is an operation execution unit, through the task_* bus. On the other hand, if the instruction is a tensor data operation instruction that directs data movement between memories, it can transmit the instruction to dp_gen_core, which is responsible for generating memory addresses.

[0116] According to one embodiment, the electronic device (400) generates memory addresses required for tensor data operations through a dp_gen_core unit and can manage up to two data operations simultaneously. The dp_gen_core, upon receiving a data operation command from dp_fetch, can utilize a plurality of dp_gen modules contained within it to calculate complex memory address sequences such as the starting address, transfer size, and stride for each data transfer. The ability to simultaneously process two independent data operations (e.g., data loading from external DDR memory to core internal memory and other data loading from sram_core to core internal memory) can have the effect of shortening data preparation time and maximizing the utilization rate of hardware resources.

[0117] According to one embodiment, the electronic device (400) can establish a flexible data transfer path through a source-sink structure. The dp_source unit serves to generate a DMA read request for the source of the data operation, that is, the starting point of the data. For example, if data needs to be read from external memory, a request is made to ddr_rx via the readmaster3* bus, and if data needs to be read from within the core, the readmaster1* bus can be used. The data read by the dp_source is transferred to the dp_sink unit, and the dp_sink generates a DMA write request for the destination (sink) of the data. By flexibly combining the source and sink in this way, various data transfer scenarios, such as data exchange between cores and data movement between external memory and a scratchpad, can be efficiently supported.

[0118] According to one embodiment, the electronic device (400) can overlap operations and data transfers by utilizing two main hardware threads. For example, dp_fetch can sequentially assign received instructions to a first thread and a second thread, and each thread can be configured to use a separate internal memory page. This allows the data operation pipeline (dp_gen_core, dp_source, dp_sink) of dp_core to preload data required for the next operation of the second thread into the memory page of the second thread while the core unit performs an operation belonging to the first thread.

[0119] The number of internal components of dp_core (e.g., the number of dp_gen, dp_source) or the number of hardware threads described herein are merely examples of the present invention and are not limited thereto; they can be optimized and implemented according to the target level of parallel processing or chip area constraints.

[0121] An electronic device (400) according to one embodiment automatically analyzes the parallelism of tensor operations to dynamically optimize the utilization of a VLIW processor array according to the dimension and size of an input tensor, decomposes tensor operators into multiple micro-operations, analyzes the data locality of each micro-operation to reconstruct the order of operations so that the internal memory reuse rate is maximized, automatically determines the tiling size of an input feature map when performing convolution operations by considering the internal memory capacity and filter size, and caches halo data at tile boundaries in a scratchpad memory to minimize redundant memory access, and in matrix multiplication operations, unlike the systolic array method, divides the input matrix into blocks and allocates each VLIW processor to process independent blocks, and performs the process of reducing partial results between blocks in a tree structure to reduce communication overhead on a logarithmic scale, and when performing statistics-based operations such as batch normalization and layer normalization, allocates a dedicated accumulation register for calculating the mean and variance to each VLIW processor and partial statistics of all processors Normalization parameters can be calculated in a single pass through a dedicated reduction network that merges.

[0122] According to one embodiment, the electronic device (400) can automatically analyze the parallelism of tensor operations to dynamically optimize hardware utilization. This may be an optimization process performed primarily at the domain-specific language (DSL) compiler level. The compiler can decompose a single large tensor operator into multiple independent micro-operations by considering the dimensions and size of the input tensor and the scale of the VLIW processor array. Subsequently, the data access pattern of each micro-operation, i.e., data locality, can be analyzed to reorder operations in a sequence that maximizes the data reuse rate of internal memory. This operation reordering can bring about a fundamental optimization effect that reduces system power consumption and improves overall computational throughput by minimizing the number of data transfers to external memory.

[0123] According to one embodiment, the electronic device (400) can automatically apply a tiling technique that divides and processes the input feature map into tile units when performing convolution operations. Since it is impossible to load the entire large input feature map into the small internal memory of the VLIW processor at once, the optimal tile size can be automatically determined by considering the internal memory capacity and the size of the convolution filter. At this time, data of adjacent tiles, i.e., halo data, is required to compute the boundary region of the tile. Instead of redundantly reading this halo data from external memory every time, the electronic device (400) can read it once and then cache it in the on-chip scratchpad memory for reuse. This can significantly alleviate memory bandwidth bottlenecks occurring in Convolutional Neural Network (CNN) operations and have the effect of maximizing computational efficiency through data reuse.

[0124] According to one embodiment, the electronic device (400) may use a flexible block-unit parallel processing method, unlike a systolic array method having a fixed data flow, when performing matrix multiplication operations. Input matrices may be divided into small blocks that fit the internal memory size, and tasks may be assigned so that each VLIW processor (PCore) performs block multiplication independently. After all PCores calculate their respective partial results, a reduction process is required to sum these partial results to derive a final result. In order to process this process efficiently, the present invention may configure communication between PCores in a tree structure to merge partial sums step by step. This tree-structured reduction method reduces the time overhead required for communication in a logarithmic scale (log N) rather than a linear scale proportional to the number of processors (N), thereby having the effect of maintaining high scalability even as the size of the processor array increases.

[0125] According to one embodiment, the electronic device (400) may be equipped with dedicated hardware for accelerating statistical-based operations such as batch normalization or layer normalization. While these operations may require multiple data scans to calculate the mean and variance of the data, the present invention may process this in a single pass. Each VLIW processor may have a dedicated accumulation register for accumulating partial sums and sums of squares required for the calculation of the mean and variance. Once all processors have completed partial statistical calculations for their respective allocated data, these values ​​can be merged at high speed through a dedicated reduction network implemented in hardware. This can have the effect of significantly improving the training and inference performance of the latest AI models by enabling the calculation of normalization parameters, which would otherwise require multiple software steps, to be completed in a single operation flow through hardware support.

[0126] The decomposition method of micro-operations, the tiling size determination algorithm, the allocation of block multiplication, and the structure of the reduction network described herein are merely examples and are not limited thereto; they may vary depending on the detailed specifications of the hardware being implemented or the optimization goals.

[0128] FIG. 10 is a block diagram illustrating in detail the internal architecture of a tensor arithmetic execution unit (core) of an electronic device (400) according to one embodiment.

[0129] Referring to FIG. 10, the core unit of the electronic device (400) may be a core execution engine that substantially handles all tensor computing operations (tensor operator execution tasks). The core unit receives data (dp_read*, dp_write*) and execution commands (task*) from an external dp_core and may be composed of a plurality of VLIW processor cores (pcore) that perform actual operations internally, a master processor (instr) that controls them, and a stream processor (stream) that performs data conversion in the data input / output path.

[0130] According to one embodiment, the electronic device (400) can perform large-scale parallel operations by configuring a plurality of pcores in an array form. Each pcore may be a lightweight vector processor based on a Very Long Instruction Word (VLIW) architecture. Additionally, for the efficiency of physical implementation, a plurality of pcores (e.g., 4) may be grouped into a single cell. This cell structure helps manage the electrical load (fan-out) of the instruction bus or control signals transmitted from the instr unit to all pcores and ensures signal integrity, thereby enabling stable high-speed operation.

[0131] According to one embodiment, the electronic device (400) can synchronize and control the operation of all pcores through an instr unit. The instr unit acts as a master processor for the entire pcore array and can fetch VLIW instruction codes stored in instr.rom and broadcast them equally to all pcores. Accordingly, all pcores receive the same instruction every clock cycle and perform fully synchronized operations in a lock-step manner. This allows them to operate like a massive single SIMD (Single Instruction, Multiple Data) engine, thereby maximizing data-level parallelism and simplifying control logic.

[0132] According to one embodiment, the electronic device (400) can implement hardware multithreading through an instr.instr_fetch unit and hide instruction pipeline latency. The instr.instr_fetch unit receives requests for tensor operator execution from dp_core via the task_* bus and can be responsible for scheduling 16 hardware execution threads embedded in each pcore. Although VLIW instructions have a very deep pipeline and may result in long latency until a single instruction is completed, the instr.fetch unit can keep all stages of the pipeline running at all times by feeding instructions from other threads into the pipeline in a round-robin manner every clock. This enables a high effective throughput of processing one VLIW instruction per clock per pcore, thereby maximizing the utilization of the hardware's computational performance.

[0133] According to one embodiment, the electronic device (400) can handle data movement and transformation simultaneously by placing a stream processor in the data input / output path. Data can pass through the stream processor before it is written to the core internal memory via the dp_write* bus and immediately after it is read out from the internal memory via the dp_read* bus. This stream processor can perform any mapping between input data and output data, and can process non-linear transformation operations in-line during the data movement process, such as applying an activation function in AI operations. This allows data transformation to be completed without allocating a separate operation cycle, thereby having the effect of reducing the execution time of the entire workload.

[0134] The number of pcores, the configuration method of cells, the number of hardware threads, and the VLIW instruction set stored in ROM described herein are merely examples of the present invention and are not limited thereto, and can be flexibly changed according to the target computational performance and the design specifications of the chip.

[0136] An electronic device (400) according to one embodiment executes tensor computing operations in parallel through a processor array composed of a plurality of VLIW (Very Long Instruction Word) processors, and schedules 16 hardware threads in each VLIW processor in a round-robin manner so that each stage of the instruction execution pipeline is occupied by different threads, and completely hides pipeline latency by processing instructions of different threads at each pipeline stage in an instruction pipeline with a maximum of 14 clock depths, thereby achieving a VLIW instruction throughput of one VLIW instruction per clock per processor, and controls to complete complex tensor operations in a single clock without memory stop cycles by simultaneously performing address calculation for two input parameters and one output parameter, vector data fetching, 32-bit accumulator value loading, integer operations, vector operations, and result storage within a single VLIW instruction.

[0137] According to one embodiment, the electronic device (400) may adopt hardware multi-threading technology to conceal latency occurring in a deep instruction pipeline. The instruction pipeline of a VLIW processor (PCore) may have a depth of up to 14 clocks or more to process complex operations at once, which may mean that it takes a long time for a single instruction to be completed from a single-thread perspective. To prevent the computational unit from becoming idle during this latency, each VLIW processor may have 16 independent hardware execution threads. A scheduler may feed instructions from different threads into the pipeline in a round-robin manner every clock cycle. As a result, each stage of the pipeline is always filled with instructions from different threads, thereby achieving a high throughput as if the processor completes one VLIW instruction every clock cycle. This can convert the latency of individual instructions into the throughput of the entire system, thereby maximizing the utilization of computational resources.

[0138] According to one embodiment, an electronic device (400) can perform multiple micro-operations simultaneously within a single instruction by utilizing the characteristics of a Very Long Instruction Word (VLIW) architecture. A VLIW instruction is, as the name suggests, a single instruction packet having a very long bit length, and this packet may contain multiple independent control fields for controlling different operation units. For example, during one clock cycle in which a single VLIW instruction is executed, the IALU calculates the memory address of an input / output operand, the register file interface fetches vector data from internal memory, loads a 32-bit accumulator value from the X register file, the ALU array performs vector operations, and the result of the operation can be stored back in the register bank. By processing the entire process from address calculation and data movement to actual operation in parallel at once, significantly higher computational efficiency can be achieved compared to a general RISC processor that must execute multiple instructions sequentially. Since all operations are performed in internal memory, complex tensor operations can be completed with a single clock throughput without memory stall cycles.

[0139] The number of threads, pipeline depth, round-robin scheduling method, and types and combinations of micro-operations included in a single VLIW instruction of the VLIW processor described herein are merely examples of the invention and are not limited thereto, and can be designed in various ways depending on the target performance level and power efficiency.

[0141] FIG. 11 is a block diagram illustrating the detailed configuration of an arithmetic logic unit (ALU) included inside a VLIW processor core (pcore) of an electronic device (400) according to one embodiment.

[0142] Referring to FIG. 11, the ALU of the electronic device (400) may include a datapath that performs arithmetic operations that are the core of tensor operations. The ALU can be designed concisely to optimize resource usage and focus on linear arithmetic operations essential for AI and vision processing tasks. By having complex non-linear operations (e.g., activation functions) processed by the stream processor described earlier and the ALU dedicated to high-performance linear algebra operations, a synergy can be achieved to efficiently process a wide range of algorithms throughout the system.

[0143] According to one embodiment, the ALU of the electronic device (400) may have a data path optimized for a Fused-Multiply-Add (FMA) operation. This may be for the purpose of efficiently processing operations of the form (a * b) + c, which are the basic units of convolution and matrix multiplication operations of an artificial neural network. During the operation, two 12-bit input values ​​(x1_in, x2_in) may be input into a multiplier from the internal memory (register_file) of the pcore to perform multiplication. At the same time, a 32-bit accumulator value (xreg_in) containing the accumulated result of a previous operation may be input into an adder from xregister_file. The adder may generate a new 32-bit accumulated result by adding the multiplication result of the multiplier and the xreg_in value. This structure may provide higher performance and accuracy than performing multiplication and addition separately.

[0144] According to one embodiment, the ALU of the electronic device (400) may include a shifter and a comparator for subsequent processing of the accumulated result. The shifter may perform a bit shift operation on the 32-bit result value from the adder. The direction and distance (magnitude) of the shift may be controlled via an external constant (xscalar_in) input, which can be used to efficiently process operations of multiplying or dividing an integer by a power of 2 for scaling in a quantized model. The shifted 32-bit result may be stored back in xregister_file via y_out and used for the next accumulation. At the same time, the comparator can compare this result value with 0 (e.g., GT, LT, EQ, etc.) and output the result to y2_out, and the output result can be used to generate VMASK register values ​​for conditional execution. According to one embodiment, the electronic device (400) may be equipped with a saturator for converting a high-precision operation result into a low-precision data type. Data conversion may be required to store the operation result accumulated in 32 bits back into a 12-bit vector form in register_file. At this time, an overflow or underflow may occur in which the 32-bit value exceeds the range that 12 bits can represent. In such cases, the saturator performs saturation arithmetic to clamp the result value to the maximum value if it is greater than the maximum value, and to the minimum value if it is smaller than the minimum value. This value, which is output as y3_out, can play an essential role in ensuring numerical stability, especially in quantized neural networks, by preventing wrap-around errors that occur in general arithmetic.

[0145] The number of bits of the input and output data of the ALU described herein (e.g., 12 bits, 32 bits), the types and combinations of arithmetic blocks are merely examples of the present invention and are not limited thereto; they may be modified and implemented according to the precision requirements of the target AI model or hardware resource constraints.

[0146] According to one embodiment, an electronic device (400) can support efficient simultaneous access by multiple threads by optimizing the data storage structure of a register_bank that implements the internal memory of a VLIW processor core (pcore). The internal memory of the pcore is a space used by 16 hardware execution threads and may contain two types of data: private data independent of each thread and shared data shared by all threads. In this case, the memory words for each thread's private data are not stored physically contiguously, but may be stored in an interleaved memory manner where words from different threads appear alternately. This has the effect of preventing contention caused by concentrated access to a specific memory bank when multiple threads simultaneously access their own private data. On the other hand, shared data that all threads must access in common can be placed in a contiguous address space to increase data locality and optimize access efficiency.

[0147] According to one embodiment, the electronic device (400) may include sub-components for systematically managing the decoding and execution delivery process of VLIW instructions. VLIW instructions delivered from the master processor (instr) of the core unit may be interpreted by instr_decoder2 inside pcore. This decoder may separate various micro-operations, such as vector operations, integer operations, and memory access, included in the VLIW instruction packet and generate respective control signals. The decoded control signals are delivered to the instr_dispatch2 unit, and this dispatch unit may deliver the signals to an appropriate execution unit, such as an ALU bank or register_bank, to perform actual operations and control data movement. This separate structure of the decoder and dispatch facilitates the pipeline processing of complex VLIW instructions and clarifies the role of each stage, thereby facilitating design and verification.

[0148] According to one embodiment, the electronic device (400) independently provides an ialu dedicated to integer operations and a corresponding iregister_file within a VLIW processor (pcore), thereby clearly separating control flow operations and data processing operations at the hardware level. The iregister_file may include general-purpose integer registers (e.g., R0-R7) for storing scalar integer values ​​such as a loop counter, array index, and memory pointer. The ialu can use these register values ​​to perform operations that manage the flow of a program, such as address calculation and loop control. In particular, the ialu can directly manipulate the value of the VMASK register to dynamically enable or disable a vector lane according to specific conditions. By separating the ialu dedicated to control logic from the alu for vector operations in this way, control-related operations can be prevented from being mixed into complex data paths, and the efficiency and predictability of the pipeline can be increased.

[0149] The data storage method of the register_bank described herein, the internal structure of the instruction decoder, and the types of integer operations supported by ialu are merely examples of the present invention and are not limited thereto, and may vary depending on the detailed specifications of the implemented VLIW architecture or optimization goals.

[0151] According to one embodiment, the electronic device (400) can maximize processing efficiency by dynamically reconfiguring hardware resources according to the characteristics of tensor operations. This can be implemented by utilizing Partial Reconfiguration technology, which is a unique characteristic of open source FPGA platforms. For example, an AI workload (e.g., CNN, Transformer) to be executed by the electronic device (400) can be profiled in advance to determine whether the workload is convolution-centric or matrix multiplication-centric. Based on the analysis results, the RISC-V host can dynamically overlay an array of VLIW processors (PCore) or data paths optimized for the corresponding operation in a specific area of ​​the FPGA fabric at runtime. This provides the effect of creating custom hardware for the AI ​​model in real time, thereby achieving significantly higher performance and energy efficiency for specific workloads compared to ASICs or GPUs with fixed hardware structures.

[0152] According to one embodiment, the electronic device (400) may subdivide the types of tensor operations and configure heterogeneous VLIW processor cores optimized for different operation characteristics to be mixed within an array. This may be a method of distributing tasks according to the characteristics of the operation (e.g., computation-bound, memory-bound) instead of processing all operations with a core of the same structure. For example, within a core unit, a general VLIW processor (pcore) may be placed together with a tensor operation core (pcore-T) that has an enhanced adder tree structure specialized for large-scale matrix multiplication and a vector operation core (pcore-V) that has a built-in function unit specialized for element-wise operations or non-linear function processing. In this case, the central control unit dp_core can perform the role of an intelligent task dispatcher that analyzes tensor instructions and assigns computationally intensive operations to pcore-T and vector operations with frequent memory access to pcore-V. This can selectively combine the advantages of the prior art into the VLIW architecture of the present invention, thereby providing the effect of consistent high performance for various AI models.

[0153] According to one embodiment, the electronic device (400) may introduce an AI-based predictive data prefetching mechanism to maximize the efficiency of tensor data operations. This may go beyond simply pre-fetching data required for the next operation sequentially, and may involve learning and predicting complex memory access patterns occurring in the sequence of tensor operations. For example, a lightweight Recurrent Neural Network (RNN) model mounted on a RISC-V host or a dedicated hardware block may monitor the sequence of memory request addresses of dp_core in real time. This model learns past access patterns (e.g., strides, indirect references) to predict the address of data that will be needed in the future with high accuracy and can request data transfer from the DMA controller much earlier than typical prefetching. This prediction-based streaming can more effectively conceal long latency in external memory, and the effect of reducing the time the core unit waits for data loading to near zero can be expected.

[0154] The dynamic reconfiguration method of the FPGA described herein, the specific architecture of the heterogeneous VLIW processor core, and the types of AI models used for predictive prefetching are merely examples of the present invention and are not limited thereto; they can be implemented with various modifications depending on the target performance of the system or the advancement of technology.

[0156] The examples described above may be implemented using hardware and software components or combinations thereof. For example, the mentioned devices, methods, and components may be implemented through general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or other devices capable of executing and responding to instructions. A processing unit may execute an operating system (OS) and one or more software applications running on that operating system. Additionally, depending on the execution of the software, this unit may perform functions to access, store, manipulate, process, and generate data. For the sake of understanding, a processing unit may be described as a single unit, but a person of ordinary knowledge in the art will recognize that it may include multiple processing units or various types of processing units. For example, a processing unit may consist of multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, may be used.

[0157] Software may include computer programs, code, instructions, or a combination thereof, which may configure a processing unit in a desired manner or command it independently or collectively. Software and data may be interpreted by the processing unit or may be permanently or temporarily implemented on various machines, components, physical or virtual devices, computer storage media, or transmission signals to provide instructions and data. Software may be stored or executed in a distributed manner on computer systems connected to a network and may be stored on one or more computer-readable recording media.

[0158] Although the examples above have been illustrated with specific drawings, experts in the field may apply various technical modifications and variations based on them. For instance, appropriate results can be obtained even if the described techniques are performed differently from the presented order, or if the components of the described systems, structures, devices, or circuits are combined in different forms or replaced with other components or equivalents.

Claims

Claim 1 An electronic device for processing tensor computation workloads based on an open source FPGA platform, comprising: a memory for storing instructions; and includes a processor, wherein when the instructions are executed by the processor, the electronic device receives tensor instructions on an FPGA platform based on an open-source instruction set architecture, classifies and processes the received tensor instructions into tensor data operations and tensor computing operations, performs tensor data operations for data movement between external memory and internal memory, executes tensor computing operations using tensor operators in internal memory, hides hardware complexity and executes user algorithms through Tensor Virtual Machine software for AI workload processing, receives tensor instructions from a RISC-V host via an AXILite bus through a DP core, which is the central control unit of an FPGA platform based on a RISC-V instruction set architecture, divides the tensor instructions received from the DP core into 16 hardware threads, creates registers for each thread, and secures instruction execution space, and when performing tensor data operations, reads data from external DDR memory via DDR-RX, performs data mapping in a stream processor, and then in the register bank of internal memory When performing tensor computing operations, it utilizes multiple PCores configured as VLIW (Very Long Instruction Word) processor arrays to execute parallel operations on data stored in internal memory, wherein each PCore includes 8 ALUs (Arithmetic Logic Units) to simultaneously perform arithmetic operations on 8 elements of a vector word, stores int32 accumulator values ​​in the X register file within the PCore to process fused multiply-accumulator operations in the form of (aXb)+c (FMA) in a single cycle, stores integer values ​​for loop counters, array indices, and pointer references in the I register file, and performs integer operations through IALU,An electronic device comprising a register bank composed of two internal memory pages, each page connected to one of two hardware threads to support nested execution, allowing one thread to perform memory transfers while the other thread performs operations; storing temporary data during tensor data transfer in an SRAM core acting as scratch memory; controlling the transfer of tensor operation results to external memory via DDR-TX or retention in internal memory for the next operation; and converting high-level tensor operation code written by a user into hardware-executable VLIW instructions via a Domain Specific Language (DSL) compiler, thereby enabling the same software to be recompiled and ported to different hardware while concealing hardware details. Claim 2 delete Claim 3 In claim 1, when the instructions are executed by the processor, the electronic device monitors the execution pattern of tensor operations at runtime to identify frequently used operation sequences, dynamically compiles the identified sequences into fused instructions and stores them in the instruction cache, stores the indices and values ​​of non-zero elements in a compressed form to accelerate sparse tensor operations, and performs conditional execution by utilizing the VMASK register of the VLIW processor to skip operations on zero elements, analyzes the precision requirements of tensor operations to optimize energy efficiency, applies approximate computing to small values ​​below a threshold, and verifies in real time whether the degradation of the operation result quality is within an acceptable range, assigns a context ID to each tensor instruction to support a multi-tenant environment, schedules instructions of different contexts to share hardware resources in time-sharing, but verifies access rights per memory page in hardware to ensure data isolation between contexts, and, to support debugging and profiling, the execution cycle, memory access pattern, and pipeline of each VLIW processor An electronic device that measures the frequency of stall occurrences using a hardware counter and records it in a trace buffer to enable real-time identification of performance bottlenecks, and controls critical operations for fault tolerance by redundantly executing them on two or more VLIW processors and comparing the results to provide a mechanism for re-executing in the event of a discrepancy or selecting the correct result by majority voting.

Images