A scalable hardware architecture template for processing streaming input data

A scalable hardware architecture template addresses inefficiencies in processing streaming input data by generating design parameters that adapt to varying frame sizes and arrival rates, enhancing throughput and reducing power consumption in edge devices.

JP2024544114AActive Publication Date: 2024-11-28GOOGLE LLC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024519431
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2021-12-10
Publication Date
2024-11-28
Estimated Expiration
2041-12-10

AI Technical Summary

Technical Problem

Existing hardware architectures for processing streaming input data, such as machine learning processors, face inefficiencies in handling varying frame sizes and arrival rates, leading to backpressure and increased power consumption, especially in edge devices with limited resources.

Method used

A scalable hardware architecture template is used to generate design parameters based on streaming input data characteristics, allowing for the instantiation of hardware components that efficiently process data with varying frame sizes and arrival rates, reducing backpressure and power consumption.

Benefits of technology

The solution enables hardware components to process streaming input data with high throughput and efficiency, using fewer resources and minimizing power consumption while adapting to varying data characteristics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544114000001_ABST
    Figure 2024544114000001_ABST
Patent Text Reader

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for generating an enhanced hardware architecture for processing streaming input data are described. In one aspect, the method includes receiving data representing a hardware architecture template. The hardware architecture template includes a set of configurable design parameters. Values ​​for the set of design parameters are determined based on characteristics of the streaming input data. The determination process includes generating a plurality of candidate hardware architectures based on a search space of the set of configurable design parameters, each candidate hardware architecture including a respective design parameter value, determining a respective performance value associated with each candidate hardware architecture, selecting a hardware architecture based on the respective performance values, and determining a value based on the parameter value associated with the selected candidate hardware architecture. Output data including the value is generated for instantiating a hardware architecture using the hardware architecture template.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Technical Field This specification relates to using scalable hardware architecture templates to generate hardware design parameters for hardware components (e.g., machine learning processors) that perform operations on streaming input data, and to manufacturing the processors using the parameters. [Background technology]

[0002] background Artificial intelligence (AI) is the intelligence demonstrated by a machine and describes the ability of a computer program or machine to think and learn. One or more computers can be used to perform AI calculations to train the machine for a particular task. AI calculations can include calculations represented by one or more machine learning models.

[0003] Neural networks belong to the subfield of machine learning models. A neural network may use one or more layers of nodes that represent multiple operations (e.g., vector and matrix operations). One or more computers may be configured to execute the operations or calculations of the neural network to generate an output, e.g., a classification, prediction, or segmentation of a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of network parameters. Summary of the Invention [Problem to be solved by the invention]

[0004] overview The techniques described in the following specification relate to generating hardware design parameters for hardware components (e.g., machine learning processors) that perform operations on streaming input data using a scalable hardware architecture template, and manufacturing the processor using the parameters. The hardware architecture template can include a set of configurable design parameters for manufacturing hardware components that can be configured to perform operations on streaming input data such that the architecture can be scaled up or down based on characteristics of the streaming input data. The techniques can be used to determine values ​​for the set of design parameters and instantiate a hardware architecture using the hardware architecture template and the determined values.

[0005] A hardware architecture, also referred to as a hardware architecture representation, generally relates to a representation of an electronic or electromechanical hardware block, component, or system that is designed (or to be designed). A hardware architecture may include data for identifying, prototyping, and / or manufacturing such a hardware block, component, or system. A hardware architecture may be encoded with data that represents the structure of the block, component, or system, e.g., data that identifies sub-components or subsystems included in the hardware block, component, or system and their interrelationships. A hardware architecture may also include data that represents a manufacturing process for the hardware block, component, or system, or data that represents a discipline for effectively implementing the design of the hardware block, component, and / or system.

[0006] The term "hardware architecture template" in this document refers to data representing a template having a set of design parameters for hardware components, such as machine learning processors, configured to perform machine learning computations on streaming inputs. A hardware architecture template can be a pre-configured general design of a hardware architecture with aspects that are tuned or customized based on a set of design parameters, such as, for example, the type, amount, tier, etc., of various computing devices included in the hardware architecture.

[0007] A hardware architecture template may be abstracted and not instantiated until values ​​for a set of design parameters are determined. After values ​​for the design parameters are determined, for example using various processes described herein, the hardware architecture template may be used to instantiate a hardware architecture based on the values ​​for the set of design parameters that are determined. In some implementations, the hardware architecture template may represent data that can be synthesized into a hardware circuit and encoded in a high-level computer language that can be programmed in an object-oriented manner, for example, C or C++. For clarity, the term "hardware architecture template" may also be referred to as "template" in this document.

[0008] The set of design parameters may form or have a multi-dimensional "search space" in which a search is performed for values ​​for each of the set of design parameters given certain design requirements or criteria. The values ​​of the design parameters may be determined by exploring the search space using one or more algorithms or techniques. In this document, the term "search space" refers to a solution space that encompasses all, or at least a set, of possible solutions (e.g., values) for the set of design parameters given available resources, e.g., all possible types and amounts of different computing devices included in a hardware architecture.

[0009] The templates can be reconfigured based on characteristics of the data used to perform the computational operations. In some circumstances, the hardware architecture generated by the template can be re-instantiated on the fly with changes in the input data (e.g., different input matrices for different sparse levels, etc.).

[0010] The term "hardware component" refers to a hardware component for performing computing operations (e.g., machine learning computations), including suitable hardware computing devices or clusters of computing devices configured to perform, for example, vector reduction, tensor multiplication, basic arithmetic operations, and logical operations based on streaming input data. For example, the hardware component may include one or more tiles (e.g., multiply-accumulate (MAC) units), one or more processing elements including multiple MAC units, one or more clusters including multiple processing elements and processing units, such as graphics processing units (GPUs) and tensor processing units (TPUs).

[0011] The term "streaming input data" refers to data that is continuously provided to a hardware component for processing the data. For example, the data may include multiple frames of data, where each frame is generated at a particular time interval, and where each frame of data is provided to a hardware component for processing at a particular rate. The terms "time interval" and "rate" refer to the period or frequency for generating or receiving a frame of data and the next frame. For example, the rate of the streaming input data may be one frame of data every few milliseconds, seconds, minutes, or other suitable period.

[0012] The streaming input data may be streaming image frames or video frames collected by an image sensor according to a time series. The image sensor may include a camera or a recorder. The streaming image frames may be collected by the image sensor at a particular rate or provided to a hardware component at a particular arrival rate.

[0013] Each frame of the streaming input data may have a particular size. For example, each of the streaming image frames may include a respective image resolution, such as 50×50 pixels, 640×480 pixels, 1440×1080 pixels, or 4096×2160 pixels.

[0014] The hardware components can be configured to process streaming input data received at a particular rate. As discussed above, the streaming input data can be generated continuously, e.g., frame by frame, from one or more sources and provided to the hardware components at a particular arrival rate. For example, the rate can be frames per unit time, or pixels per unit time. Ideally, the hardware components can process each frame of streaming input data to generate output data in time before the next frame of input data arrives. However, if the hardware components are unable to process a frame before the next frame arrives, the hardware components can cause backpressure to process subsequent frames of streaming input data. Backpressure can cause interruptions or time delays in output data generation, increase system overhead, especially if other hardware components in the system are configured to process the output data generated by the hardware components, or cause errors in the operation of the hardware components and / or computations by the hardware components.

[0015] In some implementations, the system can generate output data with greater precision using new streaming input data having a larger frame size and / or a higher frequency (e.g., higher resolution per image and more image frames per unit time). Hardware components that were initially adequate may no longer be able to process each frame of the new streaming input data before the next frame arrives, resulting in backpressure to process the later arriving frames of streaming input data.

[0016] Techniques for performing generalized matrix multiplication (GEMM) and generalized matrix vector multiplication (GEMV) are not applicable to processing streaming input data because each frame of the streaming input data is received consecutively. For example, each frame of the streaming input data can be represented by an input matrix, which is received row-by-row by a hardware component during a particular time window. An example of a GEMM or GEMV technique is known as loop tiling, also called loop nesting optimization, which divides the iteration space of a loop into smaller chunks or blocks to perform matrix-matrix or matrix-vector calculations, allowing each smaller chunk or block of input to be calculated in parallel. However, loop tiling techniques may not be compatible with processing streaming input data because the input is received row-by-row in a sequence. It is not possible, or at least impractical, to pre-store the last row of the current frame or the rows of the next frame and perform operations on these rows while processing another row of the current frame in parallel.

[0017] Some techniques resort to solving the backpressure problem by including more processing elements (PEs) or computing devices when the size or frequency of streaming input data increases. However, this can be inefficient and non-scalable, and can quickly reach maximum power requirements of the hardware components as frame sizes or arrival rates increase. For example, edge devices (e.g., smartphones, tablets, laptops, watches) configured to process streaming input data (e.g., perform calculations using each frame of input data) may have upper limits on their power consumption rates. Thus, the total amount or number of computing devices integrated within the hardware components of an edge device may be limited by maximum power requirements, or by battery life requirements per charge, or both.

[0018] To process streaming input data at high throughput more efficiently and robustly, the techniques described in this document implement a hardware architecture template with a set of design parameters. A system performing the described techniques can determine values ​​for the set of design parameters based on characteristics of the streaming input data and instantiate a hardware architecture using the hardware architecture template with the determined design parameter values. The hardware architecture includes a particular arrangement of computing devices specified by the design parameter values ​​and represents hardware components suitable for processing the streaming input data. The hardware architecture can be used to manufacture the hardware components.

[0019] According to one aspect, this document describes a method for generating a hardware architecture based on specific streaming input data. The hardware architecture can be used to manufacture hardware components that are sufficient to process the specific streaming input data. The method includes receiving data representing a hardware architecture template that includes a set of configurable design parameters, where the set of design parameters can include two or more of: an amount of clusters, an amount of processing units in each cluster, and a size of a hardware device array in each processing unit.

[0020] The method further includes determining values ​​of a set of configurable design parameters based at least in part on characteristics of the streaming input data processed by the hardware components. The determination process includes generating a plurality of candidate hardware architectures using a search space of the configurable design parameters, determining respective values ​​of a set of performance measures associated with each candidate hardware architecture, selecting one candidate hardware architecture from among all of the plurality of candidate hardware architectures based at least in part on the respective values ​​of the set of performance measures, and determining values ​​of the design parameters based on the selected candidate hardware architecture.

[0021] Output data generated by the method may include at least design parameter values ​​for manufacturing the hardware architecture.

[0022] In some implementations, the method includes providing the output data to a hardware architecture template, instantiating a hardware architecture based on the determined design parameter values, and manufacturing hardware components using the hardware architecture.

[0023] In some implementations, the characteristics of the streaming input data may include an arrival rate of each frame and a size of each frame. The set of performance measurements may include at least one measurement of latency, power consumption, resource usage, or throughput for processing each streaming input data of a particular hardware component. The performance model may include at least one of an analytical cost model, a machine learning cost model, or a hardware simulation model. The streaming input data may be streaming image frames collected by an image sensor according to a time series. The characteristics of the streaming image frames may include a particular arrival rate, and each frame of the streaming image frames may have a respective image resolution. In some implementations, the characteristics of the streaming image frames may include a respective image resolution of the image frames. The features of the streaming image frames may include a blanking period (e.g., a vertical blanking period and / or a horizontal blanking period), a pixel or color format (e.g., an RGB or YUV color format), and an arrival order of the image frames. The streaming input data may be streaming audio collected by an audio sensor. The characteristics of the audio streaming data may include at least one of a particular sample rate of the streaming audio, a bit depth of the streaming audio, a bit rate of the streaming audio, or an audio format of the streaming audio.

[0024] In some embodiments, the streaming input data may be received in a matrix or vector format, and the method further includes segmenting the input frame from a matrix into a plurality of vectors, decomposing the matrix into a plurality of vectors by matrix multiplication, determining a sparsity level of the matrix (e.g., a matrix stored in a memory device and used to multiply with the streaming input data), and / or determining non-zero values ​​in the stored matrix to improve computational efficiency.

[0025] Particular embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: The techniques described in this document may be robust to generate hardware components (e.g., machine learning processors) capable of processing different streaming data with different frame sizes and arrival rates. More specifically, a system executing the described techniques may customize a hardware architecture for a particular streaming input data by determining design parameter values ​​of a hardware architecture template. This technique allows for rapid determination of parameter values, thus enabling agile hardware development. The use of hardware architecture templates allows for instantiation of a hardware architecture based on the determined design parameter values, enabling scalable and customizable hardware architectures that can support streaming input data with a wide variation in data rates, data sizes, and / or other characteristics. The instantiated hardware architecture may be enhanced to reduce or even eliminate backpressure when processing a particular streaming input data. The hardware architecture may be configured to be re-instantiated on the fly to process a variety of matrices with different sparseness levels, up to a non-streaming 50% sparseness level.

[0026] Additionally, the techniques described in this document improve the efficiency of processing streaming input data. More specifically, the techniques described can use fewer computational resources, less power, and less memory to perform computations, such as machine learning computations, on the streaming input data. The design parameter values ​​of the template are determined based on one or more factors, requirements, or criteria, e.g., the design parameter values ​​can be determined to minimize power usage and maintain a particular input arrival rate. For example, the design parameters can be determined to allow the streaming input data to be processed without backpressure while meeting the power and / or size requirements of the hardware components. A system that performs the described techniques can also perform certain processing on sparse matrices to reduce memory usage. For example, the system can refrain from storing zero values ​​of non-streaming matrices to process the streaming input data or perform operations only on input values ​​associated with non-zero values ​​of the non-streaming matrices, thereby reducing computational resources to perform operations and reducing memory bandwidth for data transfer and memory size for storage.

[0027] Additionally, the techniques described in this document can process streaming input data with high throughput and performance. The techniques described can reduce latency in processing streaming input data by balancing processing speed and computing device idle time according to various processing requirements. For example, a hardware component generated by a template can process each frame of streaming input data faster, which may increase the idle time of one or more computing devices in the hardware component. Alternatively, the hardware component can process each frame at a reduced speed, but still process each frame in a timely manner. The techniques described can also ensure high throughput by avoiding potential logic congestion or hardware clock rate slowdowns. For example, the techniques described can explore only a subset of a set of design parameters until the generated hardware architecture reaches a scalability limit, where further increasing the values ​​of the subset of design parameters may cause logic congestion or adversely affect the hardware clock rate.

[0028] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 illustrates an exemplary architecture design system. [Diagram 2] FIG. 2 illustrates an example scenario for processing frames of streaming input data. [Diagram 3] FIG. 2 illustrates another example scenario for processing frames of streaming input data. [Figure 4] FIG. 2 illustrates another example scenario for processing frames of streaming input data. [Diagram 5] FIG. 2 illustrates another example scenario for processing frames of streaming input data. [Figure 6] FIG. 1 illustrates an example of data access patterns for a non-streaming matrix. [Figure 7] FIG. 1 is an example process for processing sparse non-streaming matrices. [Figure 8] FIG. 13 is a flow diagram of an example process for generating output data using a hardware architecture template. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description 1 illustrates an exemplary architecture design system 100. Architecture design system 100 is an example of a system implemented on one or more computers at one or more locations, and may implement the systems, components, and techniques described below. Some of the components of architecture design system 100 may be implemented as computer programs configured to run on one or more computers.

[0031] As shown in FIG. 1, the architecture design system 100 can include an architecture enhancement subsystem 120 configured to process input data 110 and generate output data 170 associated with an enhanced hardware architecture of hardware components.

[0032] More specifically, the output data 170 can be used to instantiate a hardware architecture that can be used to manufacture hardware components configured to process streaming input data, e.g., streaming of images. The hardware components can be configured to perform various operations to process the streaming input data, e.g., machine learning computations that use matrices or vectors stored by the components and the streaming input data. The streaming input data can be in vector, matrix, or tensor format. The hardware components can be a graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or another suitable processing unit or circuitry configured to appropriately process streaming of images.

[0033] As an example, the hardware component may be part of a client device or an edge device, such as a smartphone, computer, portable tablet, etc., and is designed with one or more computational devices configured to process streaming input data (e.g., streaming images and videos). The streaming input data may be received by the hardware component frame by frame at a defined time interval and processed by the edge device according to the received order, e.g., using other data stored in the edge device. For example, the edge device may perform inference operations of a neural network to process the input video frame by frame and recognize faces using the network weights stored in the edge device.

[0034] The input data 110 may include data representing characteristics of the streaming input data to be processed by a hardware component having a particular hardware architecture. The characteristics may include a particular receiving rate of the streaming input data. For example, the receiving rate may be one frame per millisecond, second, minute, or other suitable unit of time. In some implementations, the streaming input data may include multiple image frames, such as video. The characteristics may further include a particular data size of each frame received in a time step. For example, if each frame is an image frame, the data size may be a pixel resolution of 720×480 pixels, 1280×720 pixels, 1920×1080 pixels, 4096×2160 pixels, or more. In another example, the data size of each frame may be the amount of bits or bytes of each frame. For example, if the frames are other types of data, the data size may be expressed in bits or bytes.

[0035] The input data may also include other characteristics. One example of a characteristic may include a blanking period of a sensor configured to receive streaming input data. The blanking period may include a vertical blanking period, a horizontal blanking period, or both. A blanking period generally refers to the period between the time the sensor receives the end of the last visible line (e.g., the bottom or left line) of a frame or field and the time the sensor receives the start of the first visible line (e.g., the top or right line) of the next frame. In one particular example, the frequency (i.e., the reciprocal of the period) of the blanking period may be 60 Hz for the vertical blanking period and 15,750 Hz for the horizontal blanking period. Other frequencies may also be used. Thus, the processing rate of the hardware components is ideally capable of accommodating the blanking periods of the streaming image frames.

[0036] Another example characteristic may include the pixel format (or color format) of the streaming input image data, e.g., RGB or YUV. Additionally, the characteristics of the streaming input data may include the order of arrival of each frame of the streaming input data.

[0037] The streaming input data may be audio data or signals. For example, audio data may include one or more speech recordings uttered by one or more individuals, sound recordings, background noise, or other suitable types of audio data. Streaming audio data may include audio captured by a smart speaker or other type of digital assistant device. Streaming audio input may include podcasts, radio broadcasts, and / or other types of audio that may be captured by an audio sensor (e.g., a microphone).

[0038] The streaming input data may include different characteristics of the streaming audio input data. For example, the characteristics of the streaming audio may include a sampling rate. The sampling rate generally refers to the sampling frequency of an analog signal sampled from an audio signal using an audio sensor, i.e., the amount of sample analog signals collected per unit time. The sampling rate may be 44.1 kHz, 48 kHz, 88.2 kHz, 96 kHz, 192 kHz, or more. As another example, the characteristics of the streaming audio input data may include a bit depth. The bit depth typically refers to the bit size per audio sample, and is sometimes also referred to as the audio resolution of the audio sample. The bit depth may be 4 bits, 16 bits, 24 bits, 64 bits, or any other suitable bit depth. In some implementations, the characteristics of the streaming audio input data may include a bit rate. The bit rate generally refers to the amount of bits transmitted or processed per unit time. The bit rate can be calculated based on the sampling rate and bit depth, for example, a digital audio Compact Disc Audio (CD) has a sampling rate of 44.1 kHz, a bit rate of 1.4 Mbit / s, a bit depth of 16 bits, and double tracks. The processing rate of the hardware components is ideally faster than the bit rate of the audio streaming input data to avoid backlogs when the hardware components process the streaming input audio.

[0039] Other characteristics of the audio streaming input data may include the audio format of the data. For example, the audio streaming input data may be encoded in an audio format such as Pulse Code Modulation (PCM), MPEG-1 Audio Layer 3 (MP3), Windows Media Audio (WMA), or any other suitable audio format.

[0040] In some implementations, the input data 110 may include streaming input data that is processed by a hardware component (e.g., a machine learning processor that performs machine learning calculations). The architecture extension subsystem 120 may be configured to analyze the streaming input data to generate data that represents characteristics of the streaming input data (e.g., a receiving or arrival rate of each frame and a size of each frame).

[0041] Optionally, the input data 110 may also include data representing initial values ​​of a set of configurable design parameters for instantiating a hardware architecture template. The initial values ​​may be used to instantiate a default architecture (e.g., an architecture including one MAC device per cluster). For example, the default architecture may include a static random access memory (SRAM)-based line buffer device in a cluster, where the line buffer device has a single memory bank and is configured to store an entire line of input pixels for each frame. As another example, the initial values ​​may include data indicating a zero accumulator array for the default architecture.

[0042] While the streaming input data in the above example is a stream of image frames, it should be appreciated that the streaming input data may include various types of data, such as audio recordings, data structures such as vectors and tensors, to name a few examples.

[0043] The output data 170 may include at least a set of expansion parameter values ​​for instantiating or re-instantiating a hardware architecture using the architecture template. The set of expansion parameter values ​​is determined for a set of design parameters of the architecture template. The design parameters may include at least an amount of clusters in the hardware architecture, an amount of PEs in each cluster, a size of a MAC array in each processing element (PE), or any combination of two or more of these parameters. For example, the size of the MAC array may be 1, 4, 10, or more. As another example, the amount of PEs in each cluster may be 1, 4, 7, 20, 50, or more. As another example, the amount of clusters in the hardware architecture may be 1, 2, 8, 15, 30, or more. In some implementations, the output data 170 may include data defining the expanded hardware architecture, including the set of expansion parameter values, and any other data defining how the hardware components are to be manufactured.

[0044] The output data 170 may be coded in a high-level computer language, such as C or C++, that can be synthesized into hardware circuits and programmed in an object-oriented manner, as described above. In another example, the output data 170 may be a list of extended parameter values.

[0045] The output data 170 may be provided to a manufacturing system 175 to produce hardware components having a hardware architecture instantiated by the template using the parameter values ​​in the output data. The manufacturing system 175 may be any suitable system for manufacturing hardware components, such as a fabrication system or a chemical mechanical polishing system.

[0046] The architecture extension subsystem 120 may include an expansion engine 130 configured to generate the output data 170 by processing an architecture template 195 based on the input data 110. For example, the architecture extension subsystem 120 may include a memory device 190 configured to store and provide data representing the architecture template 195 to the expansion engine 130. Alternatively, the expansion engine 130 may receive the architecture template 195 from a server or memory device external to the architecture extension subsystem 120.

[0047] The architecture template 195 may be a high-level program code with multiple configurable design parameters. The architecture template 195 is configured to receive a set of design parameter values ​​and, when executed by the system, may generate output data representing a hardware architecture used to manufacture a hardware component for processing a particular type of streaming input data. For example, the expansion engine 130 may provide the architecture template 195 with multiple sets of design parameter values ​​to generate multiple candidate architectures 145.

[0048] The expansion engine 130 includes a candidate generator 140 configured to generate a plurality of candidate architectures 145. The candidate generator 140 can process the input data 110 and the architecture template 195 to generate the plurality of candidate architectures 145. The candidate generator 140 is configured to explore a plurality of parameter values ​​in a search space formed by a set of design parameters given available resources in a particular time period. The search space can have a size ranging from tens, hundreds, tens of thousands of design points (e.g., tuples each including respective values ​​of all design parameters), or any other suitable number of design points, depending on the targeted computational requirements for processing the streaming input data. For each set of candidate design parameter values ​​resulting from the exploration, the candidate generator 140 can instantiate a corresponding hardware architecture using the architecture template 195. Details of the exploration of the search space are described below.

[0049] The expansion engine 130 also includes an analysis engine 150 configured to analyze the candidate architectures 145 and generate a performance value 155 for each candidate architecture 145 using one or more performance models 185. For example, the performance value may include any suitable numerical value (e.g., a scalar value ranging from 0 to 100) indicative of the performance of the candidate architecture 145 in processing streaming input data. For example, the performance value 155 of the candidate architecture 145 may indicate how efficient the candidate hardware architecture 145 is when used to process streaming input data. For example, the efficiency may be based on the computation speed, the percentage of time in a backpressure situation, the data processing rate, or the power consumption or space consumption compared to the data processing rate of an architecture that meets the data processing rate requirements to avoid backpressure. It is not uncommon for a hardware architecture to be predicted to have a high performance value (e.g., 90 out of 100) when processing a first streaming input data, but a lower performance value (e.g., 30 out of 100) when processing a second streaming input data that has different characteristics than the first streaming input data. Thus, by generating performance values ​​associated with multiple, e.g., all, candidate architectures for processing a particular streaming input data, system 100 can use architecture template 195 to efficiently obtain one or more of the best performing candidate architecture designs for processing the particular streaming input data.

[0050] The performance model 185 may be an analytical model, a machine learning based model, or a simulation model configured to access various aspects of the performance of a hardware architecture for processing a particular type of streaming input data. The performance metrics may measure various aspects of the hardware architecture, such as power consumption, resource usage, throughput, or whether backpressure will be present when processing streaming input data having the characteristics exhibited by the input data 110.

[0051] The performance model 185 may be represented by data stored in a memory device 190 within the architecture enforcement subsystem 120, or may be provided by an external memory device or server.

[0052] The selection engine 160 may be configured to select a candidate architecture from the plurality of candidate architectures 145 as the extended hardware architecture based on a performance value 155, as shown in FIG. 1. For example, the selection engine 160 may select the candidate architecture with the highest performance value 155 as the extended candidate architecture. As another example, the selection engine 160 may select the candidate architecture that has a performance value 155 that exceeds a specified, e.g., predefined, threshold and uses minimal power and / or resources. For example, the selection engine 160 may filter each candidate architecture from the candidate architectures 145 that has a performance value 155 that does not meet or exceed a specified threshold. The selection engine 160 may then select a particular candidate architecture from the remaining candidate architectures based on its performance value, power consumption estimate, required resources and / or space on a circuit board, etc. For example, the selection engine 160 may select the remaining candidate architecture 145 that consumes the least power and / or requires the least space.

[0053] In another example, the selection engine 160 may filter the candidate architectures 145 based on power consumption and / or space required. For example, the device for which the hardware components are being designed may have limited available power and / or space, such as, for example, particularly if the device is a smartphone or other mobile device. In this example, the selection engine 160 may filter from the candidate architectures 145 each candidate architecture 145 that would exceed the available power or space. The selection engine 160 may then select from the remaining candidate architectures 145 based on the performance value 155, for example, by selecting the remaining candidate architecture with the highest performance value 155.

[0054] The selection engine 160 can encode the extended hardware architecture, or the extended parameter values, or both, into the output data 170 for further processing. For example, the extended parameter values ​​can be provided to multiple computers to instantiate the extended hardware architecture in parallel. As another example, the extended hardware architecture can be provided to one or more manufacturing devices for, e.g., in parallel, manufacturing corresponding hardware components based on the extended hardware architecture.

[0055] 2-5 illustrate example scenarios in which example hardware components having different designs process frames of streaming input data. For convenience, the above processes are described as being performed by one or more computer hardware components located in one or more locations. For example, hardware components manufactured using the architecture design system 100 of FIG. 1 and appropriately programmed may perform these processes.

[0056] The described hardware components manufactured using the template are configured to process streaming input data at different levels of design. For example, the hardware architecture can have a first level design of clusters, a second level design of processing elements (also referred to as processing units in this document), and a third level design of hardware device arrays (hardware computing device arrays, or hereinafter also referred to as hardware computing arrays, e.g., MAC device arrays). The described hardware architecture can be instantiated from the template after determining each level of design. For example, the design parameters can include the amount and / or placement of clusters, the amount and / or placement of PEs in each cluster, and / or the amount of hardware device arrays in each PE. As another example, the design parameters correspond to the dimensions of each hardware device array, e.g., the dimensions or amount of hardware devices (e.g., MAC devices) in the hardware device array.

[0057] As shown in FIG. 2, the exemplary hardware architecture 200 may include one cluster 230 including one processing unit 240. The processing unit 240 may include a hardware computing device array 250. As another example, in the hardware architecture 300 shown in FIG. 3, each cluster 330 may include multiple processing units 340a-c, each processing unit 340a-c having one hardware device array 350a-c, respectively. Furthermore, another example of the hardware architecture 400 may include multiple clusters 430a, 430b. Each cluster 430a and 430b may include processing units 440a and 440b. Each processing unit 440a and 440b may include one hardware device array 450a and 450b, respectively. Furthermore, another example of the hardware architecture 500 may include multiple clusters 530a-x, each cluster having multiple processing units 540a-x, each processing unit 540a-x having a hardware device array 550a-z. For ease of illustration, only two, three, or four clusters, processing units, or hardware device arrays are shown in FIGS. 2-5 for each hardware architecture 200-500, but it should be understood that the hardware architectures may include other amounts of clusters, processing units, and hardware device arrays.

[0058] The hardware architecture can be configured to process frames of streaming input data per unit of time, e.g., at a time step of each frame. Each frame of streaming input data can be received in the form of a vector with multiple dimensions (e.g., a vector with 2, 5, 10, or 20 entries). The dimensions of the input vector are 1×input dim Alternatively, each frame of streaming input data can be received in matrix form and processed as a vector by splitting the input matrix into multiple vectors.

[0059] In general, a hardware architecture can perform operations on input vectors using pre-stored matrices or vectors. dim ×output dim In some implementations, the operations involve vector or matrix multiplication, so that the output data produced by the hardware architecture is 1×output dim The output can be in vector form with dimensions , or the output can be in matrix form, for example if the operation involves matrix-matrix multiplication.

[0060] After determining a hardware architecture based on design parameter values ​​using the described template, a hardware component or system manufactured based on the described hardware architecture may divide each frame of streaming input data (e.g., an input vector) into one or more partial tiles based on the dimension of the hardware device array (e.g., the amount of MAC devices in the array). For example, assuming a MAC device array includes D MAC devices in the array, the dimension of each input tile may be dimension D. A partial tile is also referred to as a partial segment in the following specification. Each partial tile includes non-overlapping values ​​of the input vector.

[0061] Referring again to FIGS. 2-5, the streaming input may be received in matrix or vector format for each frame. If the streaming input of a frame is received in matrix format, the controller or scheduler may remap or reshape the input matrix into an elongated vector or vectors for further processing by the hardware components. For example, if each frame of streaming input data is received in matrix format, the controller or scheduler may treat each row of the matrix as a vector and convert the calculations from the matrix by matrix multiplication into vector by matrix multiplication. The other matrix multiplied by the input matrix or vector is, for example, a matrix stored in a memory device rather than additional streaming input data.

[0062] The streaming input vector 210 can be divided into multiple non-overlapping sub-segments 215a-c, each having a dimension D corresponding to the size of the hardware device array 250. A controller or scheduler (e.g., a hardware hierarchical state machine) in the system can generate these segments 215a-c and use these segments to schedule operations to be performed on different clusters, PEs, and MAC device arrays. Similarly, the streaming input vectors 310, 410, and 510 can be divided into multiple sub-segments 315a-c, 415a-c, and 515a-c, respectively. Although only three sub-segments are shown in FIGS. 2-5, it should be understood that each frame of the input vector for a time step can be divided into more than three sub-segments (e.g., 4, 8, 12, 24, 51) or another suitable number of sub-segments.

[0063] In general, the dimension D is the length of the columns or rows of the input matrix stored in the hardware component. dimFor example, the input dimension of a frame of streaming input data may be 100. Each subtile and corresponding hardware device array may have a dimension of 1, 10, 20, 50, 100, or another suitable dimension.

[0064] The component or system may store all of the partial segments in one or more buffers, for example, buffers within a processing unit that includes a hardware device array.

[0065] The hardware component or system can be configured to perform an operation on each input partial tile based on a vector of size D, which is fetched or prefetched from a corresponding row or column of a pre-stored matrix (e.g., a partial row or column corresponding to the partial tile). Referring again to FIGS. 2-5, the pre-stored matrix can be matrix data 220, 320, 420, and 520, respectively. The operation can include, for example, dot products and other suitable element-wise arithmetic operations. The hardware component or system can generate a partial output (e.g., a partial sum) by performing the above operations at this time step and store the partial output in an accumulator array, for example, accumulator arrays 260, 360, 460, and 560 shown in FIGS. 2-5, respectively.

[0066] The hardware component or system may repeatedly perform the above operations for each input sub-tile and corresponding sub-row or column of the pre-stored matrix. The total number of iterations may be based on design parameters such as, for example, the number of different clusters, PEs, hardware device arrays, and the dimension D of each hardware device array.

[0067] For example, referring again to FIG. 2, for each frame of streaming input data, a hardware component or system may perform the above operations on output dimTherefore, the size of the accumulator array to store all the partial outputs is output dim The accumulator array 260 may aggregate the stored partial outputs and provide the aggregated output for further operations.

[0068] 3, hardware architecture 300 may include multiple processing units 340a-c in cluster 330. Assuming that each processing unit 340a-c may have a hardware device array 350a-c (MAC array) of size 1 (e.g., only a single MAC device in each hardware device array 350a-c), then the amount of MAC array is equal to the amount of processing units 340a-c in cluster 330. The described hardware component or system may divide an input vector into multiple sub-tiles, each having a dimension of one element, since the dimension of the hardware device array is one.

[0069] Assuming the output dimension is equal to or greater than the amount of processing units, we can use one or more processing units to generate multiple sub-tiles, i.e.

[0070]

number

[0071] Each processor can execute one or more sub-tiles.

[0072]

number

[0073] For example, if the output dimension is 10 and the amount of processing units 350 per cluster is 5, then each processing unit 350 is used twice to process two partial input tiles each, and each processing unit 350 may have an accumulator array 360 of size 2. The processing units in FIG. 3 are designed to be equal to or smaller than the output dimension to increase efficiency of computational resources.

[0074] 4, the exemplary hardware architecture 400 may include multiple clusters, for example two clusters 430a and 430b. A streaming input vector 410 is divided into multiple sub-segments 415a-c. Each of the multiple sub-segments 415a-c has the same dimensions as the hardware device arrays 450a and 450b.

[0075] The partial segments 415a-c may be evenly distributed to each of the two clusters 430a and 430b. For example, as shown in FIG 4, the partial segments 415a and 415c are assigned to the cluster 430a, and the partial segment 415b and other partial segments (not shown) are assigned to the cluster 430b.

[0076] Each of the clusters 430a and 430b can be configured to process the partial segments assigned to it using a corresponding sub-row or sub-column of the matrix data 420. The processes and operations performed in each cluster are similar to those described with respect to FIG. 2. Each of the clusters 430a and 430b can generate a respective partial sum by processing the partial segments assigned to it, and the partial sum is 1×output dimEach cluster 430a and 430b may be further configured to provide a respective partial sum vector to an accumulator unit 455. The accumulator unit 455 may be configured to combine the partial sum vectors from different clusters to generate an output vector and provide the output vector to an accumulator array 460. In some implementations, the accumulator array may have dimensions of 1×output dim It can have dimensions.

[0077] Referring to FIG. 5, as described above, the exemplary hardware architecture 500 may include multiple clusters 530a-x, each having multiple processing units 540a-x, each having a respective hardware device array 550a-y.

[0078] Similar to the process of Figure 4, the illustrated hardware components or systems can be configured to divide a frame of input vectors into a number of partial segments 5151a-c at a time step. Each cluster 530a-x is assigned a respective subset of the partial segments substantially equally. For example, as shown in Figure 5, partial segments 515a and 515c are assigned to cluster 530a, and partial segment 515b and another partial segment (not shown) are assigned to cluster 530x.

[0079] Each cluster 530a-x performs respective processes and operations similar to those described with respect to FIG. dim Each of the inputs may generate a respective partial sum vector having a dimension of 1×output vector and provide each partial sum to an accumulator unit 555. The accumulator unit 555 is configured to combine corresponding partial sum vectors and generate an output vector that is provided to an accumulator array 560 for further operations. The accumulator array 560 has a dimension of 1×output vector. dim The dimensions may include:

[0080] Referring again to FIG. 1 and in conjunction with FIG. 2-5, the architecture design system 100 can generate hardware architectures 200, 300, 400, and 500 according to the characteristics of different streaming input data. For example, if the arrival rate of the streaming input data is slower (e.g., per second) or if the size of each frame is small (e.g., 120 pixels per image frame), the architecture design system 100 can generate a hardware architecture similar to the hardware architecture 200 using a single processing unit in a cluster. As another example, if the arrival rate of the streaming input data is faster (e.g., per millisecond) or if each frame is large in size (e.g., 4000 pixels per image frame) similar to the hardware architectures 300, 400, or 500, there are multiple processing elements in a cluster or multiple clusters.

[0081] As discussed above, the example hardware architecture may have a set of design parameter values ​​associated with at least one of a dimension of a hardware device array, an amount of hardware device arrays in processing units, an amount of processing units in clusters, and an amount of clusters in the hardware architecture. The system is configured to determine the set of design parameter values ​​using a search space formed by the set of design parameters, taking into account constraints of input data arrival rate, throughput, power consumption, and available area or space requirements. Details of determining the set of design parameter values ​​are described in connection with FIG. 8.

[0082] Turning now to pre-stored matrices used to process the input vectors, pre-stored matrices, also called non-streaming matrices, are fetched or pre-fetched into memory on the device, for example, an on-chip static random access memory (SRAM) device. The size of the pre-stored matrices scales with the size of the input vector at a time step, so larger vector inputs require larger pre-stored matrices, resulting in increased on-chip SRAM consumption.

[0083] 6 illustrates an example data access pattern for a non-streaming matrix 600. For convenience, the data access pattern is associated with a process performed by one or more computer systems at one or more locations. For example, hardware components manufactured based on a hardware architecture generated from the architecture design system 100 of FIG. 1 and appropriately programmed can execute the process that generates the data access pattern.

[0084] 5, assuming that the hardware architecture includes two clusters (e.g., clusters 630a and 630b), each cluster having three PEs (or processing units) 640a-c, and each PE2 having a MAC array of size 4, the system can be divided into two parts, shown by two rectangles, of an example non-streaming matrix 600, as shown in FIG 6. The upper part can be assigned to cluster 630a, and the lower part can be assigned to cluster 630b.

[0085] The system can access each portion of the non-streaming matrix 600 to process the corresponding partial segment. The dimension of the non-streaming matrix 600 is 8×9. For example, when the cluster 630a receives the partial segment 615a of size 4 at the PE 640a. The cluster can also access the top first column (e.g., the sub-sequence of the non-streaming matrix 600) and perform an element-wise operation on each element of the partial segment 615a with the corresponding element of the sub-sequence at the PE 640a to generate a first partial sum. Similarly, the cluster 630a can receive the partial segment 615a at the PE 640b, access the top second column, and perform an operation on the partial segment 615a and the top second column using the PE 640b to generate a second partial sum. The cluster 630a can receive the partial segment 615a at the PE 640c, access the top third column, and perform an operation on the partial segment 615a and the top third column using the PE 640c to generate a third partial sum. The first, second, and third partial sums can be arranged into a partial sum vector of dimension three.

[0086] PEs 640a-c may then repeat the operation by accessing the top 4th through 6th columns to generate a second partial sum vector of dimension 3, and the top 7th through 9th columns to generate a third partial sum vector of dimension 3. Cluster 630a may provide the first, second, and third partial sum vectors to an accumulator unit (e.g., accumulator unit 555 of FIG. 5) to form an intermediate partial sum vector of dimension 1×9.

[0087] Turning to the bottom of non-streaming matrix 600, cluster 630b and its corresponding PEs 640d-f can access each column in the bottom to generate another intermediate partial sum vector of dimension 1×9. In some implementations, the system can provide the two intermediate partial sum vectors as output data. Alternatively, the system can combine the partial sum vectors to generate output data of dimension 1×9.

[0088] If the frame of streaming input data is in matrix format, the system may perform operations according to a process similar to the technique described above for processing a frame of streaming input data. For example, if the input frame has dimensions M-by-K and is received row-by-row at a hardware component or system, and the non-streaming matrix has dimensions K-by-N, the system may process each row of the input matrix and load the non-streaming matrix M times.

[0089] However, when the size of the input frame is large and the non-streaming matrix is ​​a sparse matrix with a certain sparsity level (i.e., a matrix with a certain percentage of zero elements), loading or prefetching the large size non-streaming matrix multiple times is inefficient in terms of power consumption and computational resources. A technique for processing sparse non-streaming matrices is described in connection with FIG.

[0090] 7 is an exemplary process 700 for processing sparse non-streaming matrices. For convenience, the process 700 is described as being performed by one or more computer systems located at one or more locations. For example, hardware components manufactured according to a hardware architecture generated from the architecture design system 100 of FIG. 1 and appropriately programmed can perform the process 700.

[0091] The non-streaming matrices are pre-determined and stored in on-chip memory so that the system can determine the sparseness level of the matrices and the zero elements of the matrices. The sparseness level can be 10%, 20%, 50%, or any other suitable sparseness level.

[0092] In some implementations, the sparsity level may be a block sparsity ratio defined as the K non-zero elements in a block of 1×N vectors. The block sparsity ratio of a non-streaming matrix may be tuned to the respective task, such as face detection, gaze detection, or depth map generation. Since the sparsity level may be determined in advance, the hardware components or systems described herein may pre-process and compress the sparse matrix offline.

[0093] Additionally, the described techniques may also determine a segmentation size (dimension size D) for splitting the input vector based on a pre-determined sparsity level and characteristics of the streaming input data. After determining the dimension size D, the system may access the non-streaming matrix at a granularity of D elements and encode non-zero elements for each sub-column or row of the non-streaming matrix. In this manner, the described techniques may maximize utilization of the hardware device array and reduce metadata storage overhead and hardware index decoding complexity over existing compression formats (e.g., Compressed Sparse Row (CSR) format and Compressed Sparse Column (CSC) format).

[0094] As shown in FIG. 7, an exemplary non-streaming matrix (e.g., matrix data 720) includes non-zero elements 735, indicated by shaded regions, and zero elements 740, indicated by white regions. For example, each of vector data 735a-d includes four elements. The first and third elements of vector data 735a are non-zero, and the second and fourth elements of vector data 735a are zero. The first and fourth elements of vector data 735b are non-zero, and the second and third elements of vector data 735b are zero. The second and third elements of vector data 735c are non-zero, and the first and fourth elements of vector data 735c are zero.

[0095] The system can process each vector data 735a-d to generate respective compressed data 750a-d, each compressed data including only non-zero elements with an identifier 760 indicating a relative position with respect to the original vector data 735a-d. The identifier 760 can be generated based on an index mapping or a bitmap. After receiving the partial segment at the PE, the system can use the identifier to select a value from the partial segment to process the partial segment. The value selected from the partial segment corresponds to the non-zero element of the corresponding compressed data 750a-d.

[0096] For example, compressed data 750 generated based on vector data 735a may include only non-zero data, i.e., the first and third elements, and an identifier 760 associated with the first and third elements. The identifier 760 is configured to indicate that the first element of compressed data 750 corresponds to the first position of vector data 735a, and the second element of compressed data 750a corresponds to the third position of vector data 735a. When processing vector data 735a with the corresponding input partial segments, the system can only select values ​​from the partial segments at the first and third positions of the input partial segments and perform element-wise operations on the selected values ​​with the corresponding non-zero elements in compressed data 750a.

[0097] Further, the described techniques can support both dense and sparse computation. More specifically, the described techniques can switch a mode in which a hardware component processes streaming input data between dense and sparse modes in response to a determination of a change in an input matrix stored in the hardware component when the hardware component is performing operations to process the streaming input data. For example, a manufactured hardware component can include a control and status register (CSR) that is used to switch a hardware component that processes streaming input data with new non-streaming data from a dense matrix mode to a sparse matrix mode in response to a determination that the new non-streaming matrix qualifies for a threshold sparse value for the sparse matrix mode. Note that the identifier is used only in the sparse matrix mode.

[0098] 8 is a flow diagram of an exemplary process 800 for generating output data using a hardware architecture template. For convenience, the process 800 is described as being performed by one or more computer systems located at one or more locations. For example, a suitably programmed system such as the architecture design system 100 of FIG. 1 can perform the process 800.

[0099] The system receives 810 data representing a hardware architecture template. As described above, the hardware architecture template includes a set of configurable design parameters and is configured to instantiate a hardware architecture based on determined design parameter values. The hardware architecture can be used to manufacture hardware components configured to process particular streaming input data. The set of design parameters includes two or more of: (i) an amount of clusters in the hardware architecture, (ii) an amount of processing units in each cluster, and (iii) a size of the hardware device array in each processing unit.

[0100] The system determines (820) values ​​for a set of configurable design parameters for a hardware architecture for manufacturing the hardware components. The value determination is based at least in part on characteristics of the respective streaming input data for the given hardware components. Details of the determination process are described in connection with steps 840-870.

[0101] The system generates output data including values ​​(830). In some implementations, the output data can include an instantiated hardware architecture generated by setting the set of configurable design parameters with the determined values ​​of the hardware template. Alternatively, the output data can include both the obtained design parameter values ​​and a corresponding hardware architecture generated based on the values ​​from the template. The system can further provide output data for manufacturing a hardware component based on the hardware architecture.

[0102] To generate values ​​for the set of configurable design parameters, the system first generates 840 multiple candidate hardware architectures based on a search space for the set of configurable design parameters. As described above, the search space is based on the set of configurable design parameters and is limited by possible parameter values ​​based on available computational resources, power consumption, and on-chip area usage. The system can generate multiple candidate hardware architectures having respective sets of design parameter values ​​among one or more sets of possible design parameter values.

[0103] The set of one or more possible design parameter values ​​can be determined using one or more different search algorithms. For example, the system can perform a random search, an exhaustive search, or a genetic search algorithm.

[0104] An example of a range of a set of design parameters for manufacturing a hardware component may be 5 clusters, 20 PEs, and 100 MAC device arrays. That is, a candidate hardware component may have an amount of clusters ranging from 1 to 5, each cluster may have an amount of PEs ranging from 1 to 20, and each PE may have 1 to 100 MAC device arrays with the respective size. The system may generate multiple candidate hardware architectures using the search algorithm described above, searching multiple possible values ​​from the range of examples, and applying each set to instantiate the respective hardware component using the template. For example, the system may start with the minimum value of the set of design parameters and gradually increase the value of one or more design parameters. The system may stop searching once it obtains a set of design parameter values ​​suitable for the throughput requirements.

[0105] In some implementations, the system can explore parameter values ​​associated with the size of the hardware device array, the amount of hardware device arrays in the PE, and the amount of PEs in the cluster, but cannot explore or increase the amount of clusters until it determines a tipping point, where further increasing the size of the hardware device array or the amount of PEs per cluster would adversely affect the computation clock rate or cause logic congestion. That is, the size of the hardware device array and the amount of processing units per cluster have reached the limit of the scalability of the cluster. In this way, the system can place more hardware devices and PEs and minimize the amount of clusters to instantiate a hardware architecture that meets the required throughput.

[0106] The system determines (850) respective values ​​of a set of performance measures for each of a plurality of candidate hardware architectures. Respective values ​​of the set of performance values ​​are determined for each candidate hardware architecture using a performance model (or cost model). Each performance value is associated with a numerical value that represents a cost or a combination of multiple costs. The cost may represent a level of latency, throughput, power consumption, on-chip area usage, computational resource usage, or any suitable combination thereof.

[0107] The performance model may be any suitable model for processing a hardware architecture together with a set of design parameter values. The performance model may be an analytical model, a machine learning based model, or a hardware simulation model, just to name a few.

[0108] An analytical model can generally determine the topology of a hardware architecture (e.g., the amount of interfaces, wiring, computing devices such as multipliers, adders, logic devices, etc.) and determine a performance value of the hardware architecture based on the topology. One example of an analytical model can include a roofline-based model that generates a performance value of a hardware architecture as a function of the peak performance of the machine, the peak bandwidth of the machine, and the computational intensity. The output of the roofline-based model can be a function curve that represents an upper limit (e.g., a "ceiling") of the performance of the hardware architecture under certain computational requirements or resource limitations. The roofline-based model can automatically determine the overall performance "bottleneck" factor and output a performance value that represents a level of latency, throughput and / or power consumption, as described above.

[0109] Alternatively, the performance model may be a machine learning model (e.g., supervised learning) that is trained with labeled training examples. The training examples may be generated using high-level synthesis and register-transfer level simulation. The trained machine learning model is configured to generate predictions of performance values ​​and may be any suitable machine learning model, such as, for example, a multi-layer perceptron model.

[0110] Additionally, the performance model may be a simulation model that is capable of generating power calculations and throughput estimates based on characteristics of the hardware architecture given one or more randomized input stimuli.

[0111] The system selects 860 a candidate hardware architecture as the hardware architecture for the hardware component. More specifically, the system may select the expanded hardware architecture based at least in part on the performance value. As described above, the hardware architecture may be the candidate hardware architecture having the highest performance value. Alternatively, the hardware architecture may require the least amount of computing resources while having a suitable performance value.

[0112] The system determines a value based on design parameter values ​​associated with the selected candidate hardware architecture 870. The determined value may be included in output data provided to instantiate the hardware architecture using the template or used in manufacturing the hardware components.

[0113] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access memory device or a serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), which is generated to encode information for transmission to an appropriate receiving device for execution by the data processing device.

[0114] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. The apparatus may also be or further include special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an environment for the execution of computer programs, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0115] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, such as a standalone program or as a module, component, subroutine, or other device suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer, or on multiple computers located at one site, or distributed across multiple sites and interconnected by a data communications network.

[0116] A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. A system of one or more computer programs configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation or action.

[0117] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides an output distinct from the input. An engine may be a coded function block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine may be implemented on any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smartphone, or other stationary or portable device that includes one or more processors and a computer-readable medium. Furthermore, two or more engines may be implemented on the same computing device or on different computing devices.

[0118] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs that perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0119] A computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for carrying out or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special purpose logic circuitry. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operably connected to receive data from, transfer data to, or both of these mass storage devices. However, a computer need not be equipped with such devices. Furthermore, a computer may also be incorporated in other devices, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as, for example, a universal serial bus (USB) flash drive, to name just a few.

[0120] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks (e.g., internal hard disks and removable disks), magneto-optical disks, CD-ROM disks and DVD-ROM disks.

[0121] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer with a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device, such as a mouse, trackball, or a presence-sensing display, or other surface through which a user can provide input to the computer. Other types of devices may be used to provide for user interaction, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, tactile feedback, etc. Input from the user may be received in any form, including acoustic, speech, or tactile input, etc. Additionally, the computer may interact with the user by sending and receiving documents to and from a device used by the user, such as by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and receiving a response message in return from the user.

[0122] An embodiment of the subject matter described herein can be implemented in a computing system including a back-end component, e.g., as a data server, or including a middleware component, e.g., an application server, or including a front-end component, e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0123] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, e.g., for the purpose of displaying the data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device (e.g., results of user interaction) can be received at the server from the device.

[0124] In addition to the above-mentioned embodiments, the following embodiments are also innovative. A first embodiment of the present invention includes receiving data representing a hardware architecture template for generating a hardware architecture of a hardware component configured to perform an operation on respective streaming input data, the hardware architecture template including a set of configurable design parameters including two or more of: (i) an amount of clusters in the hardware architecture; (ii) an amount of processing units in each cluster; and (iii) a size of a hardware device array in each processing unit; and determining, for a given hardware architecture of the given hardware component, values ​​of the set of configurable design parameters based at least in part on characteristics of the respective streaming input data of the given hardware component, wherein the determining includes using the hardware architecture template to generate a plurality of candidate hardware architectures of the given hardware component based on a search space of the set of configurable design parameters. and generating output data indicative of the values ​​of the set of design parameters for the given hardware architecture. In one embodiment, the method comprises: generating candidate hardware architectures, each candidate hardware architecture including respective design parameter values ​​for a set of configurable design parameters; for each of a plurality of candidate hardware architectures, determining, based on the performance model and characteristics of the streaming input data for each of the given hardware components, respective values ​​of a set of performance measures associated with the candidate hardware architecture; selecting a candidate hardware architecture from the plurality of candidate hardware architectures as the given hardware architecture based at least in part on the respective values ​​of the set of performance measures; determining, as values ​​of the set of configurable design parameters for the given hardware architecture, design parameter values ​​associated with the selected candidate hardware architecture; and generating output data indicative of the values ​​of the set of design parameters for the given hardware architecture.

[0125] Example 2 is the method of example 1, further including: providing the output data to a hardware architecture template; instantiating the given hardware architecture based on values ​​of the set of design parameters of the given hardware architecture; and manufacturing the given hardware component based on the given hardware architecture.

[0126] Example 3 is the method of example 1 or example 2, wherein the characteristics of the respective streaming input data of the given hardware component include an arrival rate of each frame of the respective streaming input data of the given hardware component and a size of each frame.

[0127] Embodiment 4 is a method of any one of embodiments 1 to 3, wherein the set of performance measurements includes measurements of at least one of latency, power consumption, resource usage, or throughput for processing the streaming input data for each of the given hardware components, and the performance model includes at least one of an analytical cost model, a machine learning cost model, or a hardware simulation model.

[0128] A fifth embodiment is the method of any one of the first to fourth embodiments, wherein the streaming input data of each of the given hardware components includes streaming image frames collected by the image sensor according to a time sequence.

[0129] Example 6 is the method of example 5, wherein the characteristics of the streaming image frames include at least one of a particular arrival rate of the image frames and a respective image resolution of each of the image frames.

[0130] Example 7 is the method of example 5, wherein the characteristics of the streaming image frame include a blanking period including at least one of a vertical blanking period or a horizontal blanking period.

[0131] Example 8 is the method of example 5, wherein the characteristics of the streaming image frames include a pixel format, and the pixel format includes an RGB or YUV color format.

[0132] A ninth embodiment is the method of any one of the first to eighth embodiments, wherein the streaming input data of each of the given hardware components includes streaming audio collected by an audio sensor.

[0133] Embodiment 10 is a method as recited in claim 9, wherein the characteristics of the streaming input data include at least one of a particular sample rate of the streaming audio, a bit depth of the streaming audio, a bit rate of the streaming audio, or an audio format of the streaming audio.

[0134] An eleventh embodiment is a method of any one of the first to tenth embodiments, wherein performing an operation on each streaming input data using given hardware components includes: for each frame of the streaming input data, segmenting an input vector of the frame into a plurality of partial vectors, each partial vector including non-overlapping values ​​of the input vector; for each partial vector of the plurality of partial vectors, assigning the partial vector to a respective cluster of a plurality of clusters, each having a respective amount of processing units, each processing unit having a hardware device array of a respective size corresponding to values ​​of a set of design parameters of the given hardware architecture; multiplying each value of the partial vector by a corresponding value of a partial row of a matrix stored in a memory by the respective cluster to generate respective partial sums; and storing each partial sum in an accumulator array.

[0135] Example 12 is the method of example 11, wherein performing the operation on the respective streaming input data of the given hardware components using the given hardware components includes performing the operation based on a sparsity level of a matrix stored in the memory.

[0136] A thirteenth embodiment is the method according to any one of the first to twelfth embodiments, in which the operation to be performed is switched between a dense matrix mode and a sparse matrix mode, and the switching process is controlled by a control and status (CSR) register.

[0137] Example 14 is the method of example 11, wherein when generating corresponding values ​​of partial rows of a matrix stored in a memory is performed in a sparse matrix mode, the generating further includes determining non-zero values ​​of the partial rows of the matrix stored in a memory; and generating identifiers indicating positions of the non-zero values ​​of the partial rows in the matrix, the identifiers including an index or a bitmap; and generating compressed vectors of non-zero values ​​associated with the corresponding identifiers as the corresponding values ​​of the partial rows of the matrix.

[0138] Example 15 is the method of example 14, further including selecting values ​​of the partial vector corresponding to the compressed vector based on the corresponding identifier, and multiplying each of the selected values ​​of the partial vector by a corresponding non-zero value of the compressed vector.

[0139] Embodiment 16 is a method of any one of embodiments 1 to 15, wherein the given hardware architecture includes data indicating an upper limit sparseness level of one or more matrices stored in the memory, and the given hardware architecture is configured to be dynamically reinstantiated to process the streaming input data using a second matrix of the one or more matrices having a different sparseness level than a first matrix of the one or more matrices.

[0140] Embodiment 17 is a method of any one of embodiments 1 to 16, wherein generating a plurality of candidate hardware architectures using hardware architecture templates based on a search space of a set of configurable design parameters includes exploring the search space of the set of design parameters using at least one of a random search algorithm, an exhaustive search algorithm, or a genetic algorithm.

[0141] Embodiment 18 is a method of any one of embodiments 1 to 17, wherein searching the search space of the set of configurable design parameters includes searching for design parameter values ​​corresponding to the size of the hardware device array in each processing device and the amount of processing devices in the cluster, determining that the design parameter values ​​corresponding to the size of the hardware device array and the amount of processing devices in the cluster are at the scalability limit of the cluster, and accordingly searching for design parameter values ​​corresponding to the amount of the cluster.

[0142] Embodiment 19 is a system including one or more computers and one or more storage devices that store instructions operable to cause the one or more computers to perform any one of the methods of embodiments 1 to 18 when executed by the one or more computers.

[0143] Embodiment 20 is a computer storage medium encoded with a computer program, the program including instructions operable to cause the data processing device to perform any one of the methods of embodiments 1 to 18 when executed by the data processing device.

[0144] Although the present specification contains many specific implementation details, these should not be construed as limiting the scope of the invention or the scope that may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as acting in a certain combination and may initially be claimed as such, in some cases one or more features may be deleted from the claimed combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0145] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0146] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method comprising: receiving data representing a hardware architecture template for generating a hardware architecture of hardware components configured to perform operations on respective streaming input data, the hardware architecture template including a set of configurable design parameters including two or more of: (i) an amount of clusters in the hardware architecture; (ii) an amount of processing units in each cluster; and (iii) a size of a hardware device array in each processing unit; The method comprises: determining, for a given hardware architecture of a given hardware component, values ​​of the set of configurable design parameters based at least in part on characteristics of the respective streaming input data of the given hardware component, said determining including: generating a plurality of candidate hardware architectures for the given hardware component using the hardware architecture template based on a search space of the set of configurable design parameters, each candidate hardware architecture including a respective design parameter value for the set of configurable design parameters; for each of the plurality of candidate hardware architectures, determining a respective value of a set of performance measures associated with the candidate hardware architecture based on a performance model and the characteristics of the respective streaming input data of the given hardware component; selecting a candidate hardware architecture from the plurality of candidate hardware architectures as the given hardware architecture based at least in part on the respective values ​​of the set of performance measures; determining design parameter values ​​associated with the selected candidate hardware architecture as the values ​​of the set of configurable design parameters of the given hardware architecture; The method comprises: The method further comprising generating output data indicative of the values ​​of the set of design parameters for the given hardware architecture.

2. providing the output data to the hardware architecture template; instantiating the given hardware architecture based on the values ​​of the set of design parameters of the given hardware architecture; The method of claim 1 , further comprising: manufacturing the given hardware component based on the given hardware architecture.

3. The method of claim 1 , wherein the characteristics of the respective streaming input data of the given hardware component include an arrival rate of each frame and a size of each frame of the respective streaming input data of the given hardware component.

4. 2. The method of claim 1 , wherein the set of performance measurements includes at least one measurement of latency, power consumption, resource usage, or throughput for processing the respective streaming input data of the given hardware component, and the performance model includes at least one of an analytical cost model, a machine learning cost model, or a hardware simulation model.

5. The method of claim 1 , wherein the respective streaming input data for the given hardware component comprises streaming image frames collected by an image sensor according to a time sequence.

6. The method of claim 5 , wherein the characteristics of the streaming image frames include at least one of a particular arrival rate of image frames and a respective image resolution of each of the image frames.

7. The method of claim 5 , wherein the characteristics of the streaming image frame include a blanking period including at least one of a vertical blanking period or a horizontal blanking period.

8. The method of claim 5 , wherein the characteristics of the streaming image frames include a pixel format, the pixel format including an RGB or YUV color format.

9. The method of claim 1 , wherein the respective streaming input data of the given hardware component comprises streaming audio collected by an audio sensor.

10. 10. The method of claim 9, wherein the characteristics of the streaming input data include at least one of a particular sample rate of the streaming audio, a bit depth of the streaming audio, a bit rate of the streaming audio, or an audio format of the streaming audio.

11. performing an operation on the respective streaming input data using the given hardware components, For each frame of the streaming input data: segmenting the input vector of the frame into a plurality of sub-vectors, each sub-vector including unique values ​​of the input vector; For each subvector of the plurality of subvectors, assigning the sub-vectors to respective clusters of a plurality of clusters, each cluster having a respective amount of processing units, each processing unit having a hardware device array of a respective size corresponding to the values ​​of the set of design parameters for the given hardware architecture; multiplying each value of said subvector with a corresponding value of a subrow of a matrix stored in a memory by said respective cluster to generate a respective partial sum; and storing said respective partial sums in an accumulator array.

12. 12. The method of claim 11, wherein performing an operation on the respective streaming input data of the given hardware component using the given hardware component comprises performing the operation based on a sparsity level of the matrix stored in a memory.

13. 2. The method of claim 1, wherein the performing operations are switched between a dense matrix mode and a sparse matrix mode, the switching process being controlled by a control and status (CSR) register.

14. When generating the corresponding values ​​for the sub-rows of the matrix stored in memory is performed in a sparse matrix mode, the generating includes: determining non-zero values ​​for the sub-rows of the matrix stored in a memory; generating an identifier indicating a location of the non-zero values ​​of the sub-row in the matrix, the identifier comprising an index or a bitmap; The method of claim 11 , further comprising generating a compressed vector of non-zero values ​​associated with a corresponding identifier as the corresponding value of the partial row of the matrix.

15. selecting a value of the sub-vector that corresponds to the compressed vector based on the corresponding identifier; 15. The method of claim 14, further comprising multiplying each of the selected values ​​of the partial vector with a corresponding non-zero value of the compressed vector.

16. 2. The method of claim 1 , wherein the given hardware architecture includes data indicating an upper sparseness level of one or more matrices stored in memory, and the given hardware architecture is configured to be dynamically re-instantiated to process the streaming input data using a second one or more matrices having a different sparseness level than a first one of the one or more matrices.

17. 2. The method of claim 1 , wherein generating the plurality of candidate hardware architectures using the hardware architecture template based on the search space of the set of configurable design parameters comprises exploring the search space of the set of design parameters using at least one of a random search algorithm, an exhaustive search algorithm, or a genetic algorithm.

18. Exploring the search space of the set of configurable design parameters comprises: searching for design parameter values ​​corresponding to the size of a hardware device array in each processing unit and the amount of processing units in a cluster; determining that the design parameter values ​​corresponding to the size of the hardware device array and the amount of processing units in the cluster are at a scalability limit for the cluster; and correspondingly searching for design parameter values ​​that correspond to said quantity of clusters.

19. One or more computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective operations recited in any one of claims 1 to 18.

20. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform each of the operations described in any one of claims 1 to 18.

Citation Information

Patent Citations

  • Apparatus and method for generating video within a computer system

    JP1998509291A

  • Domain-specific reconfigurable logic using templates

    JP2007520795A

  • Automatic generation of streaming processor architectures

    US20060242617A1

  • Apparatus and method for processing sparse data

    US20180189032A1

  • Sparse matrix accelerator

    US20180349290A1