System and method for reducing memory requirements in neural networks

By optimizing the use of data flow and computing resources in convolutional neural networks, using counter mechanisms and rolling buffers, the problem of excessive memory demand for convolutional neural networks in high-resolution image and video processing is solved, and the embedding is achieved efficient image and video processing on devices.

CN113095493BActive Publication Date: 2024-11-12MAXIM INTEGRATED PROD INC
View PDF 16 Cites 0 Cited by

Patent Information

Application Number
CN202110023177.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-01-08
Filing Date
2021-01-08
Publication Date
2024-11-12
Estimated Expiration
2041-01-08

AI Technical Summary

Technical Problem

Existing convolutional neural networks limit the application of embedded devices due to the high memory requirements when processing high-resolution images and videos, especially in edge devices, and it is difficult to quickly and efficiently perform mathematical operations of cascading CNN layers. .

Method used

The method of generating output pixels using less than all available input data in a convolutional neural network, combined with counter mechanisms and rolling buffers, optimizes the use of data streams and computing resources, allowing more than a single neural network layer to be processed simultaneously, thereby reducing the number of times more than one neural network layer. Memory requirements and computing resources consumption.

Benefits of technology

It realizes efficient processing of high-resolution images and videos at relatively small memory sizes, reduces storage requirements, can process VGA and HD images, and improves the image and video processing capabilities of embedded devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113095493B_ABST
    Figure CN113095493B_ABST
Patent Text Reader

Abstract

Systems and methods are described herein for efficiently processing large amounts of data while performing complex neural network operations such as convolution operations and pooling operations. Given a cascade of convolutional neural network layers, various embodiments allow processing of downstream layers to begin before processing of the current or previous network layer is completed. In some embodiments, this is accomplished by utilizing a handshake mechanism or asynchronous logic to determine the active neural network layer in the neural network, and using the active neural layer to process a subset of the input data set before processing all of the content of the input data set of the first layer.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Inventors:

[0002] Mark Alan Lovell

[0003] Robert Michael Muchsel

[0004] CROSS-REFERENCE TO RELATED APPLICATIONS

[0005] This application is related to and claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Serial No. 62 / 958,666, filed on January 8, 2020, and entitled “Systems and Methods for Reducing Memory Requirements in Neural Networks,” with Mark Alan Lovell and Robert Michael Muksel as inventors. Each reference cited in this patent document is hereby incorporated by reference in its entirety and for all purposes. Background Art

[0006] A. Technical field

[0007] The present disclosure generally relates to reducing memory requirements and computational requirements in neural networks such as convolutional neural networks (CNNs). More specifically, the present disclosure relates to systems and methods for efficiently processing large amounts of data when performing complex neural network operations such as convolution operations or pooling operations.

[0008] B. Background Art

[0009] Machine learning is a subfield of artificial intelligence that enables computers to learn from examples without being explicitly programmed in the conventional sense. Some of the most exciting applications of machine learning utilize CNNs, supervised networks that can solve complex classification or regression problems, such as for image or video processing applications. CNNs use large amounts of multidimensional training data (such as images or sensor data) as input to learn salient features in these data. The trained network can be fine-tuned to learn additional features. In the inference phase, that is, after training or learning is completed, CNNs use unsupervised operations to detect or insert previously unseen features or events in new input data to classify objects or calculate outputs (such as regression). For example, a CNN model can be used to automatically determine whether an image can be classified as containing a person or an animal.

[0010] CNNs apply multiple hierarchical network layers and sublayers to an input image when making their determinations or predictions. The network layers are defined primarily by four parameters: 1) the size of the kernels (or weights), which is related to the receptive field spanned by the filter; 2) the number of kernels, which determines the number of feature maps; 3) the stride, which is the size of the step that the kernel moves with each output calculation; and 4) the amount of padding applied to the input data to produce an output map of the desired size. A characteristic of CNNs is that each network layer is used as the output of the previous layer, typically starting with the first convolutional layer and ending with one or more final layers (e.g., fully connected layers, which contain nodes whose activation values ​​convey scores). The scores indicate the likelihood that the input image can indeed be classified as containing a certain object.

[0011] A convolutional layer may use several kernels that apply a set of weights to the pixels of a convolution window of an image. The weights learned by the CNN during the training phase generate an activation value associated with that window. For each kernel, the convolutional layer may have a node (i.e., neuron) for each pixel that outputs an activation value calculated based on the set of weights. The activation value of the convolution window identifies a feature or characteristic, such as an edge that can be used to identify features elsewhere within the image.

[0012] Since all nodes of a filter can share the same set of weights, reusing weights is a common technique to improve both the utilization of weight storage space and computation time compared to fully connected neural networks (also known as multilayer perceptrons or MLPs). Unlike MLPs, where each combination of input and output pixels requires a weight, the same weights can be reused for each combination of input and output frames.

[0013] FIG1 is a general illustration of how receptive fields in a commonly used CNN save weight memory by passing the same small receptive field 104 across several data channels (or planes). Once all input data 102 of a layer are processed within a memory device (not shown) to generate the output 110 of that layer, the input data 102 in the memory device is discarded and the output data 110 is used as a new input for the next layer. In other words, the layers are processed sequentially, one at a time. Because the amount of data memory required to process relatively large images and high-resolution videos exceeds the available memory in cost-effective embedded devices, existing image or video processing applications are limited to thumbnail-sized images when using smaller embedded devices. Conventional implementations reserve a full frame's worth of buffer space for intermediate data, increasing storage requirements. As an example, processing 8-bit per channel VGA video using a CNN with 128 output channels in the first hidden layer requires 640×480×128=39,321,600 bytes (or 37.5 MiB) of intermediate storage for one layer, and even more when considering additional layers of deep neural networks (DNNs). Similarly, a 4k 8-bit HD image requires 3840×2160×128=1,061,683,200 bytes (or 1,012.5 MiB) for a single layer.

[0014] Therefore, there is a need for high-throughput methods and cost-effective embedded "at-the-edge" devices that can quickly and efficiently perform the mathematical operations inherent in cascaded CNN layers by processing more than a single neural network layer simultaneously, such as to achieve high-resolution image and video processing with a relatively small memory size. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Reference will be made to embodiments of the present invention, examples of which may be shown in the accompanying drawings. These drawings are intended to be illustrative, not restrictive. Although the present invention has been generally described in the context of these embodiments, it should be understood that the scope of the present invention is not intended to be limited to these specific embodiments. The items in the drawings are not drawn to scale.

[0016] Figure 1 is a general illustration of how receptive fields save weight memory in commonly used CNNs.

[0017] Figure 2 Methods for using two simplified CNN layers to produce output pixels using less than all available input data are presented in accordance with various embodiments of the present disclosure.

[0018] Figure 3 Shown Figure 2 A row-based exemplary implementation of the method presented in .

[0019] Figure 4 It is shown that according to various embodiments of the present disclosure, certain input data is no longer needed to generate the next output pixel of a CNN layer and can therefore be discarded to save computing resources.

[0020] Figure 5 An exemplary tier upgrade method using a counter according to various embodiments of the present disclosure is presented.

[0021] Figure 6 It is shown that three counters are used per layer according to various embodiments of the present disclosure.

[0022] Figure 7 is a flow chart of an illustrative process for efficiently processing large amounts of data when performing complex operations in a neural network in accordance with various embodiments of the present disclosure.

[0023] Figure 8 An accelerator system according to various embodiments of the present disclosure is presented.

[0024] Fig. 9 Depicted is a simplified block diagram of a computing device / information processing system in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In the following description, for the purpose of explanation, specific details are set forth in order to provide an understanding of the present invention. However, it will be apparent that those skilled in the art can practice the present invention without these details. In addition, those skilled in the art will recognize that the embodiments of the present invention described below can be implemented in various ways such as processes, devices, systems, equipment, or methods on tangible computer-readable media.

[0026] The components or modules shown in the figures illustrate exemplary embodiments of the present invention and are intended to avoid obscuring the present invention. It should also be understood that throughout this discussion, components may be described as separate functional units that may include subunits, but those skilled in the art will recognize that various components or their multiple parts may be divided into separate components or may be integrated together, including being integrated in a single system or component. It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented with software, hardware, or a combination thereof.

[0027] In addition, the connections between components or systems within the drawings are not intended to be limited to direct connections. Rather, data between these components may be modified, reformatted, or otherwise changed by intermediate components. Moreover, additional connections or fewer connections may be used. It should also be noted that the terms "coupled," "connected," or "communicatively coupled" should be understood to include direct connections, indirect connections through one or more intermediate devices, and wireless connections.

[0028] In this specification, references to "one embodiment", "preferred embodiment", "an embodiment", or "embodiments" mean that the specific features, structures, characteristics, or functions described in conjunction with the embodiment are included in at least one embodiment of the invention and may be in more than one embodiment. Moreover, the above phrases appearing in different places in this specification do not necessarily all refer to the same embodiment or embodiments.

[0029] Certain terms are used in various places in this specification for illustration and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; use of these terms may refer to a group of related services, functions, or resources that may be dispersed or aggregated.

[0030] The terms "include," "including," "comprise," and "comprising" should be understood as open-ended terms and any subsequent listed items are examples and are not intended to be limited to the listed items. Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference mentioned in this patent document is incorporated herein by reference in its entirety.

[0031] In addition, those skilled in the art should recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0032] In this document, the terms kernel, weight, filter, and activation function are used interchangeably. A "layer" includes any neural network layer known in the art.

[0033] It should be noted that although the embodiments described herein are presented in the context of layer-by-layer processing, it is contemplated that one or more mathematical equations may be used to describe the entire network to achieve the objects and advantages of the present disclosure. In particular, the present disclosure may be applied to networks utilizing an input plane-by-input plane structure.

[0034] Figure 2 Methods for using two simplified CNN layers to produce output pixels using less than all available input data are presented in accordance with various embodiments of the present disclosure. Figure 2 The convolutional layers 202, 230 in the network may use multiplication and accumulation operations. In an embodiment, the convolutional layers 202, 230 may calculate different filters 214 for each feature map, each filter 214 representing a feature common to the input data 204 (e.g., trained input data). The CNN may apply the learned filters 214 to the input data 204 to extract features such as: edges, orientations; and at a deeper level, structure and shape. In an embodiment, the pooling 240 may perform a subsampling operation to reduce the output map dimension of the convolutional layer 202. It should be understood that the convolutional layers and pooling operations may be applied any number of times, for example before testing the output of a fully connected layer (or, for classification tasks, an MLP) (not shown) to determine whether the pattern has been successfully recognized.

[0035] In detail, the convolution layer 202 applies a filtering operation to an input signal, for example, to pixels of an input image, by convolving a filter 214 on an input image 204. The filtering results are summed and output from the convolution layer 202 to the pooling 240, which performs a subsampling operation to reduce the data size. The two-dimensional convolution operation involves generating an output feature map for the current layer by using data from a 2D window in the previous layer. Each feature point in the output feature map is a weighted sum of the input data passed through a nonlinear activation function. The window is then moved by a given stride (here a stride of two) to produce the same weights reapplied (reusing the same computing circuit) to the next feature point of the new layer 230, thereby utilizing weight sharing of two or more convolution layers.

[0036] To generate Figure 2 , using only 5×5 input data 206. It should be noted that for a similar network with significantly larger input data, the same 5×5 input data 206 would be used to generate the output pixel. Thus, processing layer 230 may begin before all input data has been processed, i.e., before the entire output of the layer becomes available. Similarly, processing the third layer (not shown) may begin before the entire output of layer 230 becomes available, and so on.

[0037] In short, the neural network can generate output pixels using less than all of the input data 204 that might otherwise be available. Thus, the embodiments described herein allow the input data dimensionality to exceed the available per-channel data memory by one or more orders of magnitude. In various embodiments, this can be achieved by streaming the data into the CNN accelerator circuit, as described in reference to Figure 7 This principle is equally applicable to different types of data sources (such as audio or time series sensor data), but is easiest to understand when considering video (image) data. Embodiments of the present disclosure take advantage of the fact that for video or image data, the sensor typically scans line by line across the frame and sorts the output in the same way.

[0038] Figure 3 Shown Figure 2 For clarity, the example implementation of the method shown in FIG. Figure 2 Components similar to those shown are labeled in a similar manner. For the purpose of brevity, the description or the functions of the components are not repeated here.

[0039] In an embodiment, data is shifted, for example, from an image sensor (not shown) into a memory device (also not shown) in a row-by-row sequential manner such that a certain number of previous rows are available for processing. In an embodiment, to output pixel 262 (shown in black), input data 204 up to and including input pixel 308 is used, and input data after pixel 308 is ignored, thereby conserving computational resources that may be advantageously used to perform other or additional computations (such as processing more than one neural network layer simultaneously).

[0040] Figure 4 It is shown that according to various embodiments of the present disclosure, some input data is no longer needed to generate the next output pixel of the CNN layer and can therefore be discarded to save computing resources. Figure 3 In this manner, to generate input pixel 308, the previous data in input 204 is used, that is, all the previous data except pixel 308. In an embodiment, once output pixel 420 is generated, it is no longer necessary to Figure 4 Pixels denoted as “X” in the fill input data 304 may be filled in and their data may be discarded.

[0041] In effect, by ignoring the boundaries of CNN layers (e.g., 202, 230), more and more data from previous layers can be discarded. In an embodiment, this allows the use of a rolling buffer size that is several orders of magnitude smaller than would otherwise be required to store an entire intermediate data set (e.g., an entire image) as an intermediate result between layers. Advantageously, this enables existing hardware to process VGA-sized or HD-sized images.

[0042] Those skilled in the relevant art will appreciate that the number of discarded pixels may be specific to, for example, the type of neural network used, and may depend on the pooling stride, convolution type, etc.

[0043] Although neural networks are often organized into layers (e.g., 202, 230) that each operate on multiple input planes (or channels), it should be noted that this is primarily a convention and not a requirement for employing the systems and methods described in this document. Figure 4 In the most abstract structure of a CNN shown in , each layer (e.g., 202) can control the flow of data by asserting whether the layer needs additional data to perform a certain convolution. While equations applicable to several network layers or even the entire network can be derived, a more hardware-friendly approach is to control each layer based on subsequent layers, or to control active layers based on inactive layers. As discussed next, various embodiments disclosed herein can use a handshake mechanism or an ordering mechanism to determine which layer (or layers) is active at any given time to generate the desired output, which advantageously allows more than a single CNN layer to be processed simultaneously.

[0044] Figure 5 An exemplary layer upgrade method using counters according to various embodiments of the present disclosure is shown. In an embodiment, three counters (e.g., Figure 6 A first counter (referred to herein as a "start counter" (scnt)) can be used to count until sufficient information becomes available to start a first operation (e.g., a convolution operation or a pooling operation in a layer (e.g., layer 1)).

[0045] A second counter, referred to as the "column delta value" (dcnt), may be used to determine how much information is needed between operations (such as convolution operations) in the same row before processing reaches the end of the row. In an embodiment, this information may be derived from the number of columns to move in one direction to generate an output pixel. For example, for a 2x2 pooling operation (e.g., Figure 2 ), shifting two columns to the right produces the output pixel.

[0046] A third counter (referred to herein as a "row increment value") may be used to determine how much information is needed between rows. For example, for a 2x2 pooling operation, moving down two rows can generate a 4x4 matrix. It should be noted that in embodiments that do not utilize a pooling operation, only the start counter and the column increment value may be used without the row increment value, such as Figure 5 The situation in.

[0047] In an embodiment, these three counters can be used according to Figure 5 The process flow shown in is used to determine the active layer. This process can be implemented by using a conventional digital handshake mechanism (e.g., a push-pull mechanism between layers), or in some embodiments, can be implemented by using asynchronous logic in a local or global asynchronous controller that can control certain portions of these layers.

[0048] In an embodiment, the handshake mechanism may utilize a rolling buffer having a certain (e.g., programmable) size and using a start value, a column increment, and a row increment to facilitate handshaking and data flow at a substantially constant speed. In an embodiment, these parameters may be user defined or calculated by software and stored in registers in, for example, a controller, which may determine appropriate counts (counters) for each layer.

[0049] In an embodiment, the active layer may be switched from one layer to another depending on which layer requests (or is able to receive) data (as determined based on signaling between the layers). For example, to achieve unimpeded data flow, each active layer may notify its previous layer of its ability to receive data. Figure 5 As indicated by the example in , a set of comparators may be used to determine whether the start counter (scnt0) of layer 0 exceeds the input shift value (isval0) of layer 0, and further determine whether the column increment value (dcnt0) of layer 0 exceeds the increment shift value (dsval0) of layer 0. If so, layer 1 may become the active layer, in which multiple operations are performed, e.g., until these conditions are no longer met or the data in the buffer of processing layer 1 has been consumed, at which point the system returns processing layer 0 as the active layer, e.g., to allow the processor to perform mathematical operations on layer 0, thereby facilitating continuous data flow.

[0050] Alternatively, if at layer 1, comparator(s) determine that the start counter (scnt1) of layer 1 exceeds the input shift value (isval1) of layer 1, and the column increment value (dcnt1) of layer 1 exceeds the incremental shift value (dsval1) of layer 1, then, for example, layer 2 may become the active layer before returning to the first layer (here layer 0), and so on.

[0051] In short, the controller may determine whether sufficient data has been accumulated to perform an operation in layer n before continuing to receive additional input data or performing an operation in another layer.

[0052] In an embodiment, each layer may have its own rolling buffer(s) or intermediate buffer(s), which may be hardware implemented between network layers. However, this is not intended to limit the scope of the present disclosure, as one skilled in the art will appreciate that one buffer may, for example, serve more than one layer in sequence. One skilled in the art will further appreciate that once a buffer accumulates too much data, it may run out of available space. Conversely, if the buffer(s) are too small, undesirable underutilization of resources may result.

[0053] In an embodiment, input planes or input channels may be used as top-level structures rather than layers, depending on the connectivity of the underlying network. In an embodiment, instead of (or in addition to) layer-to-layer communication, independent input channels may communicate with each other using an asynchronous controller that uses a digital handshake mechanism to optimize the use of planes, for example, by eliminating one or more layers or allowing a plane in one layer (e.g., layer 2) to communicate with a plane in another layer (e.g., layer 0). Thus, two or more channels may be processed simultaneously in a single memory device.

[0054] It should be noted that other data structures that may have relationships with each other in a neural network can similarly be processed in a single memory device and simultaneously. Exemplary relationships include hierarchical relationships, temporal relationships, and any other relationships that connect two or more data structures to each other (e.g., one data structure is an input (or output) of another data structure, or one data structure is used to modify another data structure).

[0055] In embodiments, to enhance performance, the input data rate may be matched to the processing speed of a particular CNN, which should be at least as fast as the data source, but in embodiments may be slowed down so that the data is processed, for example, no faster (on average) than the data is delivered to the system. In embodiments, this may be accomplished by pausing or utilizing a stop mechanism that controls the flow of data through the CNN. In embodiments, the input data rate may be determined by the handshake mechanism itself, or may be programmable.

[0056] Figure 6 It is shown that three counters per layer are used according to various embodiments of the present disclosure. Figure 6 Corresponds to Figure 5 The main difference between the layer upgrade method using counters shown in Figure 6 The hierarchical relationship between the three layers is also shown to better illustrate the fallback to layer 0. A pop (input) may include data that has been received from a FIFO. However, it should be noted that various streaming implementations do not require a FIFO.

[0057] Figure 7is a flow chart of an illustrative process for efficiently processing large amounts of data when performing complex operations in a neural network according to various embodiments of the present disclosure. In an embodiment, process 700 begins when a plurality of active layers are determined using a handshake mechanism, a sequencing mechanism, or asynchronous logic.

[0058] At step 704, the plurality of active layers are used to process less than all of the content of the input data set to generate output pixels for more than one CNN layer simultaneously, the input data exceeding the per-channel data memory by at least an order of magnitude.

[0059] Finally, at step 706, the output pixels are output.

[0060] Figure 8 An accelerator system according to various embodiments of the present disclosure is shown. The accelerator system 800 includes an image sensor 802, a FIFO (e.g., 804), and a CNN 806. In an embodiment, one or more FIFOs 804 may be used, for example, one FIFO for each red, green, blue component or pixel. Other examples include a single FIFO (e.g., 804) that can be used for raw sensor values, and additional channels may be used for stereo cameras or infrared imaging.

[0061] Figure 8 The accelerator system 800 shown in is not limited to the structural details shown here or described in the accompanying text. As will be appreciated by those skilled in the art, a suitable hardware accelerator may include a controller that may be coupled between the image sensor 802 and the FIFO 804.

[0062] As will be appreciated by those skilled in the art, an efficient input mechanism can greatly improve the efficacy of the disclosed systems and methods. Therefore, in an embodiment, FIFO 804 or rolling buffer memory can be implemented as an intermediate storage device that can feed data into CNN 806, which can be appropriately trained to output a decision result or output another data stream. In an embodiment, non-input data can use a rolling buffer memory that functions like FIFO 804.

[0063] Likewise, the image sensor 802 can be configured to be directly coupled to a hardware accelerator so as to output its sample values ​​directly to the accelerator in a streaming manner (e.g., line-by-line scanning). In such a streaming implementation, the image sensor 802 can scan rows and columns sequentially, and in some embodiments, output horizontal and vertical synchronization signals, such as HREF and VSYNC. Advantageously, tightly coupling the image sensor 802 with the CNN 806 reduces communication overhead and requires little or no intermediate storage.

[0064] In an embodiment, the accelerator system 800 can autonomously obtain data from the input as needed, for example by using the FIFO 804, and stop or pause when insufficient data is available to perform the next operation. In an embodiment, the data input can utilize a controller-less bus master (not shown) that autonomously (i.e., directly) adds the image sensor data to one or more input FIFOs 804 as soon as the sensor data becomes available.

[0065] It should be appreciated that streaming input data may be particularly useful for neural networks where the output data dimensionality is reduced from one layer to another, for example when using pooling operations.

[0066] Fig. 9 900 is a simplified block diagram of an information handling system (or computing system) according to an embodiment of the present disclosure. It will be understood that the illustrated functionality of system 900 may operate to support various embodiments of the computing system, but it should be understood that the computing system may be configured in different ways and include different components, including having Fig. 9 Fewer or more components than depicted in .

[0067] like Fig. 9 As shown, computing system 900 includes one or more central processing units (CPUs) 901 that provide computing resources and control the computer. CPU 901 may be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 919 and / or floating point coprocessors for mathematical calculations. System 900 may also include system memory 902, which may take the form of random access memory (RAM), read-only memory (ROM), or both.

[0068] Multiple controllers and peripherals are also available, such as Fig. 9As shown. Input controller 903 represents an interface to various input devices 904 such as a keyboard, mouse, touch screen and / or stylus. The computing system 900 may also include a storage controller 907 for interfacing with one or more storage devices 908, each of which includes a storage medium such as a tape or disk, or an optical medium that can be used to record instruction programs of operating systems, utilities and applications, which programs may include embodiments of programs that implement various aspects of the present disclosure. (Multiple) storage devices 908 may also be used to store processed data or data to be processed according to the present disclosure. The system 900 may also include a display controller 909 for providing an interface to a display device 911, which may be a cathode ray tube (CRT), a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or other types of displays. The computing system 900 may also include one or more peripheral device controllers or interfaces 905 for one or more peripheral devices 906. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. The communication controller 914 can interface with one or more communication devices 915, which enables the system 900 to connect to remote devices via any of a variety of networks (including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN)) or via any suitable electromagnetic carrier signal (including infrared signals).

[0069] In the system shown, all major system components can be connected to bus 916, which can represent more than one physical bus. However, the various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. In addition, programs implementing various aspects of the present disclosure can be accessed from a remote location (e.g., a server) via a network. Such data and / or programs can be transmitted by any of various machine-readable media including but not limited to the following: magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specially configured for storage or for storing and executing program codes, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM devices and RAM devices.

[0070] Aspects of the present disclosure may be encoded on one or more non-transitory computer-readable media using instructions for one or more processors or processing units to enable steps to be performed. It should be noted that the one or more non-transitory computer-readable media should include volatile memory and non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. (Multiple) ASICs, programmable arrays, digital signal processing circuit systems, etc. can be used to implement the functions of hardware implementations. Therefore, the "means" term in any claim is intended to cover both software implementations and hardware implementations. Similarly, the term "one or more computer-readable media" as used herein includes software and / or hardware or a combination thereof having an instruction program embodied thereon. In view of these alternatives to the implementations, it will be understood that the drawings and accompanying descriptions provide functional information that will be needed by those skilled in the art to write program code (i.e., software) and / or manufacture circuits (i.e., hardware) that perform the required processing.

[0071] It should be noted that the embodiments of the present disclosure may further relate to a computer product having a non-transitory tangible computer-readable medium having thereon computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the present disclosure, or they may belong to the category known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, but are not limited to: magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specially configured for storing or for storing and executing program codes, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM devices and RAM devices. Examples of computer codes include machine codes such as those generated by a compiler and files containing high-level codes executed by a computer using an interpreter. The embodiments of the present disclosure may be implemented in whole or in part as machine executable instructions that may be in a program module executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in a local, remote, or both environment.

[0072] Those skilled in the art will recognize that no computing system or programming language is essential for the practice of the present disclosure. Those skilled in the art will also recognize that the multiple elements described above may be physically and / or functionally divided into multiple submodules or combined together.

[0073] It will be appreciated by those skilled in the art that the foregoing examples and embodiments are exemplary and not limiting to the scope of the present disclosure. It is intended that all permutations, enhancements, equivalents, combinations, and improvements thereto that are apparent to those skilled in the art after reading this specification and studying the drawings are included within the true spirit and scope of the present disclosure. It should also be noted that the elements in any claim may be arranged in different ways, including with a variety of dependencies, configurations, and combinations.

Claims

1. A method for processing neural network data, the method comprising: Using a counter to determine one or more active neural network layers in a neural network for processing the neural network data, wherein the counter is configured to count one or more shifts for data flow control of an input data set to determine whether to start an operation in one of the one or more active neural network layers or to switch from the active neural network layer to another of the one or more active neural network layers; using the one or more active neural network layers to process a subset of the input data set of a first neural network layer, the input data set comprising at least one of audio data or image sensor data, the data size of the subset being smaller than the size of the input data set; outputting a first output data set from the first neural network layer; using the first output data set in a second neural network layer; and outputting a second output data set from the second neural network layer before processing all of the input data set. 2 . The method of claim 1 , further comprising discarding at least some content of the subset after outputting the first output data set.

3. The method according to claim 1, wherein: The input data set is scanned row by row, and the first output data set is sorted row by row.

4. The method according to claim 1, further comprising: The one or more active neural network layers are determined using at least one of a handshaking mechanism, an ordering mechanism, or asynchronous logic.

5. A system for processing neural network data, the system comprising: processor; and a non-transitory computer-readable medium containing instructions that, when executed by the processor, cause steps to be performed, the steps comprising: using a counter to determine one or more active layers in a neural network for processing the neural network data, wherein the counter is configured to count one or more shifts for data flow control of an input data set to determine whether to start an operation in one of the one or more active layers or switch from the active layer to another of the one or more active layers; using the one or more active layers to process a subset of the input data set of a first neural network layer, the input data set comprising at least one of audio data or image sensor data, the data size of the subset being smaller than the size of the input data set; outputting a first output data set from the first neural network layer; using the first output data set in a second neural network layer; and outputting a second output data set from the second neural network layer before processing all of the input data set.

6. The system of claim 5, further comprising a rolling buffer coupled to the processor, the rolling buffer processing the subset of the input data set.

7. The system of claim 5, further comprising a convolutional neural network (CNN) accelerator circuit coupled to the processor and the sensor, the CNN accelerator circuit streaming the subset from the sensor to the processor.

8. A non-transitory computer readable medium comprising one or more sequences of instructions which, when executed by at least one processor, cause steps to be performed, the steps comprising: Using a counter to determine one or more active neural network layers in a neural network for processing neural network data, wherein the counter is configured to count one or more shifts for data flow control of an input data set to determine whether to start an operation in one of the one or more active neural network layers or switch from the active neural network layer to another of the one or more active neural network layers; using the one or more active neural network layers to process a subset of the input data set of a first neural network layer, the input data set comprising at least one of audio data or image sensor data, the data size of the subset being smaller than the size of the input data set; outputting a first output data set from the first neural network layer; using the first output data set in a second neural network layer; and outputting a second output data set from the second neural network layer before processing all of the input data set.

9. The non-transitory computer readable medium of claim 8, wherein: After outputting the first output data set, at least some contents of the subset are discarded.

10. The non-transitory computer readable medium of claim 8, wherein: The size of the subset depends on at least one of the pooling stride and the convolution type or the neural network type.

Citation Information

Patent Citations

  • Systems and methods for efficient neural network deployments

    CN106503791A

  • Convolutional neural network

    CN110050282A

  • Rapid calculation of convolutional neural network

    CN110537193A

  • Convolution neural network training apparatus and method thereof

    US20160162782A1

  • Systems and methods for optimizing operations of computing devices using deep neural networks

    US20170364792A1