Variable Clock Adaptation in Neural Network Processors

Abstract

Various embodiments of the present disclosure relate to variable clock adaptation in a neural network processor. Systems and devices are provided for improving the computational efficiency and / or power efficiency of one or more neural networks via computationally driven closed-loop dynamic clock control. A clock frequency control word is generated based on information indicating a current frame execution rate of a processing task of the neural network and a reference clock signal. A clock generator generates a clock signal for the neural network based on the clock frequency control word. A reference frequency can be used to generate the clock frequency control word, and the reference frequency can be based on information indicating the sparsity of data in a training frame.

Description

Technical Field

[0001] The present disclosure relates generally to power and computational efficiency of neural networks and, more particularly, but not exclusively, to the control of neural network clocks. Background Art

[0002] Various efforts have been made to improve the computational efficiency and / or power efficiency of various types of neural networks, including recurrent neural networks (RNNs), deep neural networks (DNNs), convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, etc. Such efficiency may be particularly important for such neural networks when implemented via mobile devices. Typically, computational efficiency can be evaluated in terms of tera operations per second (TOPS), which is a common performance metric for high-performance neural network systems. TOPS per watt (TOPS / W) expands this measurement to describe performance efficiency. In general, the higher the TOPS / W associated with a neural network system-on-chip (SOC), the more efficient the SOC. Summary of the Invention

[0003] In one embodiment, a device includes control word generation circuitry operable to generate a clock frequency control word based on information indicating a current frame execution rate of a processing task of a neural network and a reference clock signal. A clock generator coupled to the control word generation circuitry generates a neural network clock signal for use by the neural network based on the clock frequency control word.

[0004] In one embodiment, a system includes a neural network accelerator circuitry and closed-loop clock rate control circuitry coupled to the neural network accelerator circuitry. The closed-loop clock rate control circuitry generates a clock signal for the neural network accelerator circuitry and includes: a control word generation circuitry that, in operation, generates a clock frequency control word based on information indicating a current frame execution rate of a processing task of the neural network accelerator circuitry and a reference clock signal; and a clock generator coupled to the control word generation circuitry. The clock generator generates the clock signal for the neural network accelerator circuitry based on the clock frequency control word.

[0005] In one embodiment, a method includes executing a kernel using a neural network accelerator and performing closed-loop clock rate control of the operating frequency of the neural network accelerator during execution of the kernel. The closed-loop clock rate control includes generating a clock frequency control word based on information indicating a current frame execution rate of the kernel and a reference clock signal; and generating a clock signal for the neural network accelerator circuitry based on the clock frequency control word. In one embodiment, the contents of a non-transitory computer-readable medium configure the neural network accelerator system to perform one or more of the methods disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figures 1A to 1D Graphically depicts the convolution process performed by an exemplary convolutional neural network (CNN) with various degrees of neural network pruning.

[0007] Figure 2 Depicted is an exemplary CNN system including a sparsity compute rate (SCR) controller according to one or more embodiments of the techniques presented herein.

[0008] Figure 3 Depicted is a sparsity computation rate controller in accordance with one or more embodiments of the techniques presented herein.

[0009] Figure 4 is a functional block diagram of one embodiment of an electronic device or system utilizing computationally driven dynamic clock control of an exemplary neural network in accordance with the techniques described herein.

[0010] Figure 5 Depicted is an example operating routine for a sparsity computation rate controller in accordance with one or more embodiments of the techniques described herein. DETAILED DESCRIPTION

[0011] In the following description, certain details are set forth in order to provide a thorough understanding of various embodiments of devices, systems, methods, and articles. However, those skilled in the art will appreciate that other embodiments may be practiced without these details. In other cases, well-known structures and methods associated with, for example, circuits (such as transistors, integrated circuits, logic gates, memories, interfaces, bus systems, etc.) are not shown or described in detail in some of the figures to avoid unnecessarily obscuring the description of the embodiments.

[0012] Unless the context requires otherwise, throughout the following specification and claims, the word "comprise" and variations thereof (such as "include" and "comprising") should be interpreted in an open, inclusive sense, that is, as meaning "including, but not limited to." Unless the context dictates otherwise, references to "at least one of" should be interpreted to mean either or both disjunctive and inclusive.

[0013] Throughout this specification, reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout this specification are not necessarily referring to the same embodiment or all embodiments. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments to yield additional embodiments.

[0014] Headings are provided for convenience only and do not interpret the scope or meaning of the disclosure.

[0015] The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the shapes and angles of various elements are not drawn to scale, and some of these elements are enlarged and positioned to improve the readability of the drawings. Furthermore, the particular shapes of the depicted elements are not necessarily intended to convey any information about the actual shape of the particular element and are selected solely for ease of identification in the drawings.

[0016] Typical solutions for improving the power efficiency of neural networks involve: static adaptive voltage scaling (AVS), in which the voltage supplied to one or more processors of the neural network is modified based on one or more static lookup tables related to the current computational load on the one or more processors; dynamic AVS, in which closed-loop control of on-chip processing and / or variation monitoring is performed generally based on the output of one or more sensors; adaptive body biasing (ABB), such as can be used in silicon processes with fully depleted silicon-on-insulator (FDSOI) substrates; and dynamic voltage frequency scaling (DVFS), which generally utilizes lookup tables to scale the operating frequency of the neural network hardware according to the voltage when the predicted load on the one or more processors of the neural network is low.

[0017] Neural networks (including recurrent neural networks, deep neural networks, convolutional neural networks, LSTM neural networks, etc. as non-limiting examples) can utilize network pruning to reduce the computational load on the neural network (NN) itself and on any coupled neural network accelerator subsystems. In at least some embodiments of such pruning, after the neural network is trained, the neural network is then fine-tuned to determine the importance of specific connections within the neural network. In general, pruning effectively compresses the trained neural network by iteratively removing specific connections or nodes that are determined to be of less importance. In some embodiments, weighted information that is determined to be of less relative importance, such as based on secondary and / or derivative data, can also be removed.

[0018] Depending on the specific implementation, pruning can result in a significant reduction in the number of connections or nodes in a neural network, with a corresponding reduction in power requirements, while largely maintaining the accuracy of the results produced. Neural network pruning can also result in relative sparsity of data, which can allow the neural network (and / or associated neural network accelerator subsystem) to run at a lower clock speed while still handling the computational load. In particular, neural network size reduction / compression techniques (e.g., pruning based on kernel weights or activations, quantization, and range segmentation) can bring additional sparsity to the data used in neural network processing.

[0019] The techniques described herein generally relate to computationally driven dynamic clock control that utilizes information indicative of data sparsity in the closed-loop control of a phase-locked loop (PLL) or frequency-locked loop (FLL) clock generator coupled to, for example, a neural network accelerator subsystem. In certain embodiments, such computational clocks can improve energy efficiency (such as measured in TOPS / W or other suitable terms) for neural network or other processing, such as relative to a switched single-frequency clock. In at least some embodiments, the techniques described herein are compatible with down-spread frequency modulation, such as down-spread spectrum modulation that modulates below the nominal clock frequency. Furthermore, in certain embodiments, the techniques described herein (such as modulation of the system operating frequency based on data sparsity) can provide similar results to some down-spread modulation schemes, such as by operating to prevent exceeding the system's desired maximum operating frequency. Furthermore, such techniques can facilitate avoiding large-activity power perturbations and the attendant settling ripple within coupled power distribution systems. For example, by gradually modulating the power supply via a low-bandwidth control loop that can follow the average frequency transition based on the average value of data sparsity during execution, this technique can be advantageously compared to schemes involving clock gating and / or power gating to avoid such power perturbations and the resulting stability ripples.

[0020] In various embodiments, this technique can be utilized independent of specific neural network accelerator architectures and limitations, and can be used for parallel operation of a single neural network, a series of neural networks, and / or multiple neural networks, as well as in conjunction with other systems such as other processing accelerators (e.g., gated recurrent unit or "GRU" accelerators, LSTM accelerators, etc.), encoders, decoders, etc. Notably, this variable clock adaptation based on dynamic changes in, for example, data sparsity can be utilized alone or in conjunction with additional solutions for improving neural network efficiency, including the AVS, ABB, and DVFS types of solutions mentioned above (as non-limiting examples).

[0021] Figures 1A to 1D The convolution process performed by an exemplary convolutional neural network (CNN) with various degrees of pruning is graphically depicted, such as to illustrate the corresponding data sparsity resulting from such pruning. In the depicted embodiment, the exemplary CNN system 201 (see Figure 2) is intended for image frame recognition (such as video frame recognition and / or still frame recognition provided to a CNN system), it being understood that other embodiments of the techniques described herein may be used for any other purpose suitable for implementation via one or more neural networks.

[0022] Figure 1A Depicted is a series of six kernels K1 to K6 applied to six corresponding image features F1 to F6, respectively, during an image frame recognition process. Notably, here, the CNN computations are not pruned, such that there is no pruning effect on zero-valued features or kernel data. Thus, regardless of the values ​​present in a single kernel or feature, each of the six corresponding convolution applications requires computation cycles to complete, as depicted via six computation cycles 110a.

[0023] Figure 1B The same series of six kernels K1 to K6 are depicted, applied to six corresponding image features F1 to F6, respectively. However, here, the values ​​of the CNN's kernels K2 and K5 have each been weighted to zero, effectively removing from processing the application of those pruned kernels K2 and K5 to the corresponding image features F2 and F5. Thus, only four computation cycles 110b are required to process a single frame.

[0024] Figure 1C Again, the same series of six kernels K1 to K6 are depicted, applied to six corresponding image features F1 to F6, respectively. Here, in the depicted arrangement, the values ​​of the CNN's features F3 and F5 have each been weighted to zero, effectively removing from processing the application of the corresponding kernels K3 and K5 to those pruned features F3 and F5, thereby pruning the exemplary CNN. Consequently, only four computation cycles 110 c are required to process a single frame.

[0025] Figure 1D Once again, the same series of six kernels K1 to K6 are depicted, applied to six corresponding image features F1 to F6, respectively. Here, the values ​​of each of the CNN's features F3 and F5, and each of the kernels K2 and K5, have been weighted to zero, respectively, to prune the exemplary CNN in such a way as to remove from processing the application of the corresponding kernels K3 and K5 to the pruned image features F3 and F5, and the application of the pruned kernel K2 to the image feature F2. Thus, only the application of kernel K1 to image feature F1, the application of kernel K4 to image feature F4, and the application of kernel K6 to image feature F6 are performed, meaning that only three computation cycles 110c are required to process a single frame.

[0026] from Figures 1A to 1DAs can be seen in the example of , pruning the neural network can increase the sparsity of the resulting data and reduce the processing resources associated with a single image frame analyzed by the neural network, so that different output feature map data consumes different numbers of processing cycles. Therefore, in at least some embodiments of the technology described herein, the indication of start interrupts and stop interrupts associated with a single processing frame can allow real-time measurement of processing cycles and frame execution time, as well as serve as an indication of data sparsity for the neural network. It should be understood that although Figures 1A to 1D The pruning illustrated in generally describes a single cluster processing, but similar pruning during parallel cluster processing can result in similar data sparsity and can be reflected in a corresponding reduction in frame processing cycles.

[0027] Figure 2 An exemplary CNN system 201 utilizing a sparsity compute rate (SCR) controller 210 according to one embodiment of the techniques presented herein is depicted. As previously mentioned, while in the depicted embodiment, the exemplary CNN system 201 will be discussed herein as intended for use in video frame recognition and / or image frame recognition provided to the CNN system, other embodiments of the described techniques may be used for any other purpose suitable for implementation via one or more neural networks.

[0028] In the depicted embodiment, the SCR controller 210 is coupled between the PLL / FLL clock generator 212 and the neural network accelerator subsystem 214. The SCR controller 210 is coupled to a real-time clock 216. In operation, the SCR controller provides closed-loop control of the CNN clock by utilizing inputs to the SCR controller, including an indication of frame execution time received from the NN accelerator subsystem 214, such as the illustrated frame synchronization start and stop interrupts 218. These start and stop interrupts provide the SCR controller with an indication of the duration of the frame calculation. Based on this duration, the SCR controller 210 sets the frequency control word (F clk )222 is output to the PLL / FLL clock generator. In one embodiment, a downward extended frequency value can be generated, which results in low output jitter. Based on the frequency control word F clk To determine the nominal frequency of the NN clock output for the PLL / FLL clock generator.

[0029] In various embodiments, the frequency control word F clk222 can be generated in various ways. As an example, a frequency control word can be generated based on kernel sparsity, such as during an inference pass of the exemplary CNN system 201. In a typical configuration, kernel sparsity remains unchanged during execution of the CNN system regardless of the input data. Therefore, kernel sparsity can be determined at runtime, such as by processing static configuration inputs or by dynamically calculating kernel sparsity through execution parsing of a neural network for one or more complete frames. In some embodiments, the dynamic calculation of kernel sparsity can thereby provide a downward bias for the frequency control word. As another example, the frequency control word can be calculated based on activation sparsity, which in some embodiments can be a data-dependent and / or weight-dependent metric. In such an embodiment, the NN accelerator subsystem 214 can generate a frequency control word by passing the activation sparsity metric through, for example, a proportional and integral (P&I) control loop. The parameters of the P&I control loop can be selected to meet the bandwidth and / or damping requirements of the CNN system 201.

[0030] In operation of one embodiment, the first time the system is to be executed, the CNN system 201 initiates a learning phase. In the first pass of execution, in which only non-zero operations are performed, the CNN system 201 determines a target value for a nominal frequency or frames per second, which can be an indicator of the nominal sparsity of the kernel in general. Subsequently, the SCR controller 210 generates a frequency control word 222 based on the target value for the nominal frequency or frames per second and feedback indicating the current frames per second processed by the neural network accelerator subsystem 214. For example, in a particular image to be classified, one or more of the input data values ​​may be zero or below a pruning threshold and, therefore, produce a processing frame rate that is faster than the target frame rate. In this case, a control word can be generated to reduce the operating frequency of the clock provided to the neural network accelerator subsystem 214.

[0031] Frequency control word changes may also occur during a processed frame due to quality of service constraints on other shared system resources (such as global and / or external memory). In some embodiments, the frequency control word may be updated at a rate such that the modulation update rate of the frequency control word is less than the bandwidth of the coupled PLL / FLL clock generator 212.

[0032] In some embodiments, a nominal frequency or target frames per second rate can be determined periodically, which can facilitate accounting for changes in operating conditions, such as those due to temperature, fluctuating voltage, aging, and the like. In some embodiments, an initial nominal frequency or target frames per second rate can be determined, for example as discussed above, and then adjusted during operation, for example, based on an average value of the frame processing rate. The average value can be a moving average over a threshold number of frames (e.g., one hundred frames), and filtering can be employed to prevent the average value from straying too far from the initial nominal frequency or frame rate per second. The initial nominal frequency or target frame rate can be a default value, a value determined using a lookup table, and the like.

[0033] As indicated elsewhere herein, additional neural network efficiency solutions may be incorporated by controlling the PLL / FLL clock generator 212 based on an indication of frame execution time via the SCR controller 210. In the depicted embodiment, the PLL / FLL clock generator 212 is coupled to an AVS / DVFS controller 224, which may provide additional inputs to the PLL / FLL clock generator, such as scaled operating frequency information for the CNN system 201 based at least in part on an input voltage and a predicted and / or current computational load. Also in the depicted embodiment, the AVS / DVFS controller 224 is coupled to a voltage regulator 226, such as a low dropout regulator (LDO) or a DC-DC converter. In certain embodiments, for example in conjunction with a high degree of neural network pruning, the neural network clock signal 240 provided by the clock generator 212 may enable an AVS / DVFS controller or other controller, such as an embedded controller in the neural network accelerator subsystem 214, to select a lower input voltage for the coupled neural network, thereby facilitating improved power and / or operating efficiency.

[0034] In some embodiments, inputs to the SCR controller 210 can include neural network weighting factors 220. For example, in some embodiments, each of the interrupts 218 can be associated with a neural network weighting factor. In some embodiments, the SCR controller 210 can generate or be provided with a target value for a desired number of frames per second, such as via a quality of service controller 228 and an interconnect bus 230. In the depicted embodiment, the interconnect bus 230 is communicatively coupled between the NN accelerator subsystem 214 and the shared memories 232a, 232b, and 232c, and is communicatively coupled to a host 234 (e.g., a host system that controls the operation of the NN accelerator subsystem 214 (see Figure 4As another example, in some embodiments, a neural network accelerator subsystem, such as NN accelerator subsystem 214, may be shared among multiple neural networks (not shown) that are executed during a single time period (e.g., via time division multiplexing (TDM) or other speculative scheduling scheme). In such embodiments, NN weighting factors 220 may be used by the neural network accelerator subsystem to determine relative processing schedules for execution of those multiple neural networks.

[0035] Figure 3 Depicts Figure 2 As depicted, the SCR controller 210 is driven from the NN accelerator subsystem 214 (not shown for clarity). Figure 3 ) receives the start and stop interrupt information 218 to generate the frequency control word (F clk )222 to control the PLL / FLL clock generator (such as Figure 2 The SCR controller 210 includes a frequency counter 305, a frequency error generator 310, and a proportional-integral controller 315. In operation, the frequency counter 305 receives a start frame interrupt signal and a stop frame interrupt signal from the NN accelerator subsystem (and optionally, neural network weighting factors 220, which may be used as a linear gain scaler in some embodiments). The frequency error generator 310 determines or retrieves the nominal frequency (F nominal )325 and the frequency output from the frequency counter 305, and determines the difference between the frequency counter output and the nominal frequency F nominal The frequency error is provided to the proportional-integral controller 315, which scales and integrates the provided frequency error. The output of the proportional-integral controller 315 comprises the frequency control word 222, which is input to the PLL / FLL clock generator 212 ( Figure 2 ) to generate the CNN clock signal for the NN accelerator subsystem 214.

[0036] Figure 4 is a functional block diagram of an exemplary electronic device or system 400 that may utilize various embodiments described herein. System 400 may be used, for example, to implement a convolutional neural network to classify sensor data. It should be appreciated that because such neural networks can be computationally intensive, the ability to efficiently modify one or more operating frequencies based on information required by the neural network (such as data sparsity) can provide significant improvements in power management and overall performance of such neural networks. In various implementations, system 400 may include a system on a chip.

[0037] System 400 includes global memory 402, which can serve as main memory, such as for one or more neural network processes or processing clusters and for processes or processing clusters of one or more host systems 404. Global memory 402 includes memory management circuitry 406 and one or more shared memory arrays 408. In operation, memory management circuitry 406 employs one or more memory management routines to allocate regions of shared memory array 408 to various processes executed by system 400.

[0038] As illustrated, the system 400 includes one or more data movers 410, one or more memory bridges 420, one or more sensors 430 and corresponding sensor interfaces 432, one or more convolution accelerator engines 440, and one or more connected engines 450 that can be implemented and operated to produce classification output 460.

[0039] In operation, the data mover 410 moves data streams between IO (e.g., sensor interface 432), memory hierarchies (e.g., global memory 402, memory bridge 420), the convolution accelerator engine 440, and connected engines 450. In the depicted embodiment, the convolution accelerator engine 440 includes an SCR controller 442 and a clock generator 444, where the SCR controller 442 is coupled to a real-time clock 446 in the manner described elsewhere herein with reference to additional embodiments.

[0040] In some embodiments, the systems 201 , 400 and controller 210 may include more components than illustrated, may include fewer components than illustrated, may separate the illustrated components into separate components, may combine the illustrated components, etc., as well as various combinations thereof.

[0041] Figure 5 Depicts an exemplary operating routine 500 for a sparsity compute rate (SCR) controller 510 (the features and functionality of the SCR controller 510 may be substantially similar to those described above with reference to FIG. 1 ) according to one or more embodiments of the techniques described herein. Figures 2 to 3 The SCR controller 210 and / or Figure 4 442).

[0042] The operating routine 500 begins with the SCR controller 510 receiving a reference clock signal 502 , a reference frame processing frequency 504 , and frame duration information, such as the illustrated frame synchronization information 506 .

[0043] At 515, the SCR controller 510 determines the frame processing frequency based on the reference clock signal and the frame synchronization information 506. Figure 3 In the depicted embodiment, the frame synchronization information 506 includes start / stop interrupt information regarding frame data of a processing task of a neural network communicatively coupled to the SCR controller 510 .

[0044] At 520 , the SCR controller 510 determines a difference between the determined frame processing frequency and the reference frame processing frequency 504 .

[0045] At 525 , the SCR controller 510 generates a frequency control word (error signal) based on the determined difference between the determined frame processing frequency and the reference frame processing frequency.

[0046] At 530 , the SCR controller 510 generates a neural network clock signal based on the frequency control word, such as for use by a communicatively coupled neural network.

[0047] An embodiment of a method of controlling a neural network clock may include Figure 5 Additional actions not shown in the Figure 5 All actions shown in the Figure 5 The actions shown in , and can be modified in various aspects. For example, routine 500 can be modified to generate a reference frame processing frequency.

[0048] Some embodiments may take the form of or include a computer program product. For example, according to one embodiment, a computer-readable medium including a computer program is provided, the computer program being adapted to perform one or more of the methods or functions described above. The medium may be a physical storage medium, such as a read-only memory (ROM) chip, or a disk, such as a digital versatile disk (DVD-ROM), a compact disk (CD-ROM), a hard disk, a memory, a network, or a portable media article to be read by an appropriate drive or via an appropriate connection, including code encoded as one or more bar codes or other related codes stored on one or more such computer-readable media and readable by an appropriate reader device.

[0049] In addition, in some embodiments, some or all of the methods and / or functionalities may be implemented or provided in other ways, such as at least partially implemented or provided in firmware and / or hardware, such firmware and / or hardware including, but not limited to: one or more application-specific integrated circuits (ASICs), digital signal processors, discrete circuit systems, logic gates, standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc., as well as devices using RFID technology and various combinations of the above.

[0050] In one embodiment, a device includes: control word generation circuitry operable to generate a clock frequency control word based on information indicating a current frame execution rate of a processing task of a neural network and a reference clock signal; and a clock generator coupled to the control word generation circuitry and operable to generate a neural network clock signal for use by the neural network based on the clock frequency control word. In one embodiment, the control word generation circuitry determines the frequency control word based on a reference frequency. In one embodiment, the information indicating the current frame execution rate includes frame synchronization information, and the control word generation circuitry includes: a frequency counter operable to determine a current frame processing frequency based at least in part on the reference clock signal and the frame synchronization information; and a proportional-integral controller operable to generate the clock frequency control word based on a difference between the determined frame processing frequency and a reference frequency. In one embodiment, the reference frequency is based on a frame processing frequency of execution of a kernel that processes only non-zero operations. In one embodiment, the reference frequency corresponds to a nominal kernel data sparsity associated with the processing task of the neural network. In one embodiment, the reference frequency is based on an average frame execution rate. In one embodiment, the control word generation circuitry includes a frequency error generator coupled to the frequency counter, the frequency error generator being operable to determine a difference between the determined frame processing frequency and a reference frequency. In one embodiment, the device includes an adaptive voltage scaling controller that is operable to modify a supply voltage of the neural network. In one embodiment, the adaptive voltage scaling controller modifies the supply voltage based on a neural network clock signal. In one embodiment, the device includes a dynamic voltage frequency scaling controller that is operable to generate control information, and the clock generator generates the neural network clock signal based on the control information generated by the dynamic voltage frequency scaling controller. In one embodiment, the information indicating the current frame execution rate includes a plurality of start interrupt times associated with a processing task and a corresponding plurality of stop interrupt times associated with the processing task. In one embodiment, each of the plurality of start interrupt times corresponds to a start time associated with processing a corresponding data frame, and each of the plurality of stop interrupt times corresponds to a stop time associated with processing the corresponding data frame. In one embodiment, each of the plurality of start interrupt times and the plurality of stop interrupt times is associated with a processing weight, and in operation, the control word generation circuitry generates the clock frequency control word based on the processing weight. In one embodiment, the information indicating the current frame execution rate is associated with current kernel data sparsity of the processing task of the neural network. In one embodiment, the clock generator is one of the group consisting of a phase-locked loop (PLL) clock generator and a frequency-locked loop (FLL) clock generator.

[0051] In one embodiment, a system includes: a neural network accelerator circuitry; and closed-loop clock rate control circuitry coupled to the neural network accelerator circuitry and operable to generate a clock signal for the neural network accelerator circuitry, the closed-loop clock rate control circuitry comprising: a control word generation circuitry operable to generate a clock frequency control word based on information indicating a current frame execution rate of a processing task of the neural network accelerator circuitry and a reference clock signal; and a clock generator coupled to the control word generation circuitry and operable to generate the clock signal for the neural network accelerator circuitry based on the clock frequency control word. In one embodiment, the control word generation circuitry is operable to generate the clock frequency control word based on a reference frequency. In one embodiment, the system includes an adaptive voltage scaling controller operable to modify a supply voltage of the neural network. In one embodiment, the system includes a dynamic voltage frequency scaling controller operable to generate dynamic clock control information, wherein the clock generator generates the clock signal for the neural network accelerator circuitry based on the dynamic clock control information.

[0052] In one embodiment, a method includes executing a kernel using a neural network accelerator; and during execution of the kernel, performing closed-loop clock rate control of an operating frequency of the neural network accelerator, performing the closed-loop clock rate control including generating a clock frequency control word based on information indicating a current frame execution rate of the kernel and a reference clock signal; and generating a clock signal for the neural network accelerator circuitry based on the clock frequency control word. In one embodiment, the clock frequency control word is generated based on the reference frequency. In one embodiment, the method includes modifying a supply voltage of the neural network based on the generated clock signal of the neural network accelerator circuitry. In one embodiment, the method includes modifying the operating frequency of the neural network using a dynamic voltage frequency scaling controller.

[0053] The various embodiments described above can be combined to provide further embodiments. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications, and publications to provide yet further embodiments.

[0054] These and other changes can be made to the embodiments in light of the above detailed description. In general, in the claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and claims, but should be construed to include all possible embodiments and the full scope of equivalents to which such claims are entitled. Therefore, the claims are not limited by this disclosure.

Claims

1. An electronic device comprising: a control word generation circuitry operable to determine a current frame processing frequency based on frame synchronization information included in information indicating a current frame execution rate of a processing task of the neural network and a reference clock signal, and to generate a clock frequency control word based on a difference between the determined current frame processing frequency and the reference frequency; as well as a clock generator coupled to the control word generation circuitry and operable to generate a neural network clock signal for use by the neural network based on the clock frequency control word, The reference frequency is based on an average frame execution rate. The electronic device according to claim 1 , wherein the clock frequency control word is an error signal.

3. The electronic device of claim 2, wherein the control word generation circuit system comprises: a frequency counter operable to determine the current frame processing frequency based at least in part on the reference clock signal and the frame synchronization information; as well as A proportional-integral controller is operable to generate the clock frequency control word based on the determined difference between the current frame processing frequency and the reference frequency. 4 . The electronic device of claim 3 , wherein the reference frequency is based on a frame processing frequency of execution of a kernel that processes only non-zero operations.

5. The electronic device of claim 4 , wherein the reference frequency corresponds to a nominal kernel data sparsity associated with the processing task of the neural network.

6. The electronic device of claim 3, wherein the control word generation circuitry comprises a frequency error generator coupled to the frequency counter, the frequency error generator being operative to determine a difference between the determined current frame processing frequency and the reference frequency.

7. The electronic device of claim 1 , comprising an adaptive voltage scaling controller operable to modify a supply voltage of the neural network.

8. The electronic device according to claim 7, wherein: In operation, the adaptive voltage scaling controller modifies the supply voltage based on the neural network clock signal.

9. The electronic device of claim 1 , comprising a dynamic voltage frequency scaling controller that generates control information in operation, and the clock generator generates the neural network clock signal based on the control information generated by the dynamic voltage frequency scaling controller.

10. The electronic device according to claim 1, wherein the information indicating the current frame execution rate comprises: A plurality of start interrupt times associated with the processing task, and a corresponding plurality of stop interrupt times associated with the processing task.

11. The electronic device of claim 10 , wherein each of the plurality of start interrupt times corresponds to a start time associated with processing a corresponding data frame, and each of the plurality of stop interrupt times corresponds to a stop time associated with processing the corresponding data frame.

12. The electronic device of claim 11, wherein each of the plurality of start interrupt times and the plurality of stop interrupt times is associated with a processing weight, and in operation, the control word generation circuitry generates the clock frequency control word based on the processing weight.

13. The electronic device of claim 1 , wherein the information indicating the current frame execution rate is associated with current kernel data sparsity of the processing task of the neural network.

14. The electronic device according to claim 1, wherein the clock generator is one of the group consisting of a phase-locked loop (PLL) clock generator and a frequency-locked loop (FLL) clock generator.

15. An electronic system comprising: Neural network accelerator circuit system; as well as a closed-loop clock rate control circuitry coupled to the neural network accelerator circuitry and operable to generate a clock signal for the neural network accelerator circuitry, the closed-loop clock rate control circuitry comprising: a control word generation circuitry operable to determine a current frame processing frequency based on frame synchronization information included in information indicating a current frame execution rate of a processing task of the neural network accelerator circuitry and a reference clock signal, and to generate a clock frequency control word based on a difference between the determined current frame processing frequency and the reference frequency; as well as a clock generator coupled to the control word generation circuitry and operable to generate the clock signal for the neural network accelerator circuitry based on the clock frequency control word, The reference frequency is based on an average frame execution rate. The electronic system according to claim 15 , wherein the clock frequency control word is an error signal.

17. The electronic system according to claim 15, comprising: An adaptive voltage scaling controller is operable to modify a supply voltage of the neural network.

18. The electronic system according to claim 15, comprising: A dynamic voltage frequency scaling controller is configured to generate dynamic clock control information in operation, wherein the clock generator generates the clock signal of the neural network accelerator circuit system based on the dynamic clock control information.

19. A method for generating a neural network clock signal, comprising: Execute kernels using a neural network accelerator; as well as During execution of the kernel, performing closed-loop clock rate control of an operating frequency of the neural network accelerator, performing the closed-loop clock rate control comprising: determining a current frame processing frequency based on frame synchronization information included in information indicating a current frame execution rate of the core and a reference clock signal, and generating a clock frequency control word based on a difference between the determined current frame processing frequency and the reference frequency; as well as generating a clock signal for a neural network accelerator circuit system based on the clock frequency control word, The reference frequency is based on an average frame execution rate.

20. The method of claim 19, wherein the clock frequency control word is an error signal.

21. The method according to claim 19, comprising: A supply voltage for the neural network is modified based on the clock signal generated by the neural network accelerator circuitry.

22. The method of claim 19, comprising: A dynamic voltage frequency scaling controller is used to modify the operating frequency of the neural network.

Images