Data processing in pixel-to-pixel neural networks

Abstract

Techniques are provided for processing data in a neural network. An exemplary method may include processing rows of a first stripe of a data input by each layer of the neural network, where the rows are processed horizontally and consecutively according to a layer-wise sequence, processing subsequent rows in the first stripe by each layer after processing the rows on a row-by-row basis, where each subsequent row is processed horizontally and consecutively according to a layer-wise sequence, generating an output stripe based on the processing of the rows and subsequent processing, processing a second stripe of the data input by each layer, where each row in the second stripe is processed horizontally and consecutively according to a layer-wise sequence and the rows in the second stripe are processed on a row-by-row basis, and generating another output stripe based on the processing of the second stripe.

Description

[Technical field]

[0001] TECHNICAL FIELD The present disclosure relates generally to neural networks. For example, aspects of the present disclosure relate to techniques and systems for processing data in neural networks. [Background technology]

[0002] Various devices and systems may implement processing engines and networks to process various types of data and generate particular outputs and / or effects. For example, some devices may implement neural networks to process images and / or videos and generate outputs for consumption (e.g., for display on the device, for use by an application, etc.). Neural networks can effectively perform a variety of complex tasks with accurate results. The output and functions of neural networks may also be used in a variety of applications, such as, for example, computer graphics, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition, data processing (e.g., image processing, video processing, etc.), data post-processing, visual rendering, television, among others.

[0003] Neural networks often process large amounts of data, such as video, image, and / or audio data, to perform certain tasks and generate specific outputs. The amount of data processed by a neural network and the processing demands of the neural network can place a significant strain on device resources, such as memory, power, and computational resources. Furthermore, some neural network architectures may be more resource intensive than other neural network architectures and / or may require or utilize additional resources for their operation. This further increases the resource and / or processing costs of the neural network, reduces the efficiency of the neural network, and / or in some cases increases the operational and resource strain and requirements of the neural network. Summary of the Invention [Means for solving the problem]

[0004] In some examples, techniques and systems are described for increasing neural network efficiency and reducing neural network resource and processing burden and costs. According to at least one illustrative example, a method for processing data in a pixel-to-pixel neural network is provided. In some examples, the method includes the steps of obtaining input data with stripes that logically segment the input data, each stripe of the input data including at least one row of data; processing rows of data in a first stripe of the input data by each layer of the neural network, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network; and after processing the rows of data in the first stripe, processing the rows of data in the first stripe by each layer of the neural network. the neural network based on the processing of the row of data and the subsequent row of data; after processing the first stripe, processing rows of data in a second stripe of input data on a row-by-row basis by each layer of the neural network, where each row of data in the second stripe is processed horizontally in accordance with a layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0005] In accordance with at least one illustrative example, a non-transitory computer-readable medium for processing data in an inter-pixel neural network is provided. In some aspects, the non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to: obtain input data with stripes that logically segment the input data, each stripe of the input data including at least one row of data; process, by each layer of the neural network, rows of data in a first stripe of the input data, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network; and, after processing the rows of data in the first stripe, process, by each layer of the neural network, rows of data in a first stripe of the input data, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network. processing, by each layer of the neural network, subsequent rows of data in the first stripe on a row-by-row basis, where each subsequent row of data is processed successively in a horizontal, layer-by-layer sequence; generating, by the neural network, a first output stripe based on the processing of the row of data and the subsequent row of data; after processing the first stripe, processing, by each layer of the neural network, rows of data in a second stripe of the input data on a row-by-row basis, where each row of data in the second stripe is processed in a horizontal, layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0006] In accordance with at least one illustrative example, an apparatus is provided for processing data in an inter-pixel neural network. In some aspects, the apparatus may include a memory having computer-readable instructions stored thereon and one or more processors, the one or more processors including: obtaining input data with stripes that logically segment the input data, each stripe of the input data including at least one row of data; processing rows of data in a first stripe of the input data by each layer of the neural network, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network; and, after processing the rows of data in the first stripe, processing the rows of data by each layer of the neural network. processing subsequent rows of data in the first stripe by each layer of the neural network on a row-by-row basis, where each subsequent row of data is processed successively in a horizontal, layer-by-layer sequence; generating a first output stripe by the neural network based on the processing of the row of data and the subsequent row of data; after processing the first stripe, processing rows of data in a second stripe of the input data by each layer of the neural network on a row-by-row basis, where each row of data in the second stripe is processed in a horizontal, layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0007] According to another illustrative example, an apparatus for processing data in an inter-pixel neural network includes: means for acquiring input data with stripes that logically segment the input data, each stripe of the input data including at least one row of data; means for processing rows of data in a first stripe of the input data by each layer of the neural network, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network; and means for processing rows of data by each layer of the neural network after processing the rows of data in the first stripe. The method may include a step of processing subsequent rows of data in the first stripe on a row-by-row basis, where each subsequent row of data is processed successively in a horizontal, layer-by-layer sequence; generating a first output stripe by the neural network based on the processing of the row of data and the subsequent row of data; after processing the first stripe, means for processing rows of data in a second stripe of input data on a row-by-row basis, by each layer of the neural network, where each row of data in the second stripe is processed in a horizontal, layer-by-layer sequence; and means for generating a second output stripe based on the processing of the second stripe.

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may store in a first memory an output generated by a previous layer of a neural network for one or more blocks in a row of data in a first stripe, and store one or more lines of data from one or more blocks in the row of data in a second memory associated with a particular layer of the neural network. In some examples, the one or more lines of data may include a portion of a data input of a particular layer of a neural network for a previous stripe-block-row.

[0009] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may generate inputs for a particular layer of a neural network based on a combination of one or more lines of data that include outputs generated by a previous layer and portions of the particular layer's data input for a previous stripe-block-row, and may generate additional outputs by the particular layer of the neural network based on the outputs for the previous layer.

[0010] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may determine a portion of the input for a particular layer and store the portion of the input in a third memory associated with the particular layer of the neural network.

[0011] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may generate an additional input for a subsequent layer of the neural network based on a combination of the additional output of a particular layer and a portion of the input for the subsequent layer from a previous stripe-block-row of the subsequent layer, and generate a second additional output by the subsequent layer of the neural network based on the additional input for the subsequent layer.

[0012] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may store the second additional output in a fourth memory. In some examples, the second memory and the third memory may include a line store in a line store memory, and the first memory and the fourth memory may include a buffer in a scratch memory.

[0013] In some examples, horizontally sequentially processing the rows of data in the first stripe may include sequentially processing multiple blocks of data in the rows by each layer of the neural network, and in some cases, each layer of the neural network processes each preceding block of data depthwise before processing a subsequent block horizontally.

[0014] In some examples, processing the subsequent rows may include sequentially processing each of the blocks of data in each subsequent row by each layer of the neural network, where each layer of the neural network processes a preceding block of data in the subsequent row depthwise before processing a subsequent block of data in the subsequent row.

[0015] In some examples, the neural network may include a pixel-to-pixel neural network and the input data may include pixels associated with an image.

[0016] In some examples, obtaining the input data may include logically segmenting the input data into stripes, each stripe including a respective portion of the input data.

[0017] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may store in a first memory associated with a particular layer of the neural network the output generated by a previous layer of the neural network for one or more blocks in each subsequent row of data, and store in a second memory one or more lines of data from one or more blocks in each subsequent row of data. In some examples, the one or more lines of data may include one or more portions of one or more data inputs of a particular layer of the neural network.

[0018] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may generate inputs for a particular layer of a neural network based on a combination of one or more lines of data including one or more portions of the output generated by a previous layer and one or more data inputs of the particular layer for the previous stripe-block-row, and may generate additional outputs by the particular layer of the neural network based on the output from the previous layer and a subset of the inputs from the particular layer for the previous stripe-block-row.

[0019] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may determine a portion of the input for a particular layer, store the portion of the input in a third memory associated with a subsequent layer of the neural network, and store an additional output in a fourth memory. In some cases, the second memory and the fourth memory may include memory buffers.

[0020] In some examples, processing the row of data may include processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network, where the output corresponds to the row of data.

[0021] In some examples, processing subsequent rows of data may include processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network, where the output corresponds to a subsequent row of data of the first stripe.

[0022] In some embodiments, the device may be or be part of a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smartphone," or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR), an augmented reality (AR), or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a 3D scanner, a multi-camera system, or other device. In some embodiments, the device includes a camera or multiple cameras for capturing one or more images. In some embodiments, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the devices described above may include one or more sensors.

[0023] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0024] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0025] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a simplified block diagram illustrating an exemplary image processing system, according to some examples of the present disclosure. [Diagram 2] FIG. 13 illustrates example inputs to an inter-pixel neural network implementing stripe and raster based processing schemes, according to some examples of the present disclosure. [Diagram 3] FIG. 2 illustrates an exemplary data processing scheme for a neural network, in accordance with some examples of the present disclosure. [Figure 4] FIG. 2 illustrates an example memory architecture for storing stripe data for generating layer inputs, in accordance with some examples of the present disclosure. [Figure 5A] FIG. 1 illustrates an exemplary data processing architecture for generating neural network inputs and outputs, according to some examples of the present disclosure. [Figure 5B] FIG. 1 illustrates an exemplary data processing architecture for generating neural network inputs and outputs, according to some examples of the present disclosure. [Figure 6] A diagram showing example layer inputs padded with vertical line offsets and used to generate layer outputs, in accordance with some examples of the present disclosure. [Figure 7] FIG. 2 illustrates an exemplary funnel processing scheme for reducing redundant calculations when processing stripes, in accordance with various aspects of the present disclosure. [Figure 8] 1 is a flowchart illustrating an example process for processing data in a neural network in accordance with some aspects of the present disclosure. [Figure 9] FIG. 1 illustrates an example system architecture for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purposes of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0028] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0029] As mentioned above, neural networks can effectively perform various complex tasks with accurate results. Often, neural networks can process large amounts of data, such as video, image, and / or audio data, to perform some tasks and generate certain outputs. The amount of data processed by a neural network and the processing demands of the neural network can place a significant burden on device resources, such as memory, power resources, and computational resources. Some neural network architectures can be more resource intensive than other neural network architectures and / or may require or utilize additional resources for their operation. This further increases the resource and / or processing costs of the neural network, reduces the efficiency of the neural network, and / or in some cases increases the operational and resource burden and requirements of the neural network.

[0030] For example, inter-pixel neural networks (and other networks that similarly process other types of data) generally use large memory buffers to temporarily store data processed by the inter-pixel network. However, the amount of data stored in the memory buffer and / or the memory buffer requirements may increase significantly, even unsustainably, as the size of the input data increases. This may result in increased hardware costs and requirements. Resources in a device may limit the type, amount, and / or size of data that an inter-pixel neural network can process. In many cases, even larger, more expensive memory buffers may not be able to support some inter-pixel neural network implementations and / or enable an inter-pixel neural network to process certain types of data.

[0031] In some cases, the memory buffers used by the inter-pixel neural network may be oversubscribed. This may result in increased demands on other resources, such as power and computational resources. For example, oversubscription of the memory buffers used by the inter-pixel neural network may spike the bandwidth, processor clock cycles, and power used by the inter-pixel neural network. When the memory buffers are oversubscribed and / or the inter-pixel neural network is unable to store additional data in the memory buffers, performance metrics in the device may increase significantly. In some cases, such problems may be reduced or mitigated by increasing the amount and / or capabilities of the hardware resources of the device. For example, the memory resources and capabilities in the device may be increased to process the larger amount of data used by the inter-pixel neural network. However, the amount of memory resources required to process certain types of data and / or to perform certain tasks may be expensive or even cost prohibitive.

[0032] As another example, a custom processor may be added in a device to handle some or all of the operations of the inter-pixel neural network. Although the custom processor may increase the processing power in the device and support the additional processing demands of the inter-pixel neural network, the inter-pixel neural network may still require a large amount of memory to process the data. However, the memory resources required to support such a large amount of memory may be cost prohibitive and / or in some cases may prevent the implementation of a custom processor to address the memory requirements of the inter-pixel neural network. The memory and resource issues associated with the inter-pixel neural network may be exacerbated in lower-end device implementations.

[0033] For example, in lower-level devices, the amount of memory resources used by the inter-pixel neural network is more limited. In many cases, the size of the memory buffer used by the inter-pixel neural network may need to be reduced. In general, the memory system or subsystem for transferring data to and / or from the memory buffer may also need to be reduced as well. However, the reduction of the memory buffer may cause a spike in bandwidth requirements, as previously described. This may be further exacerbated by the reduction of the memory system or subsystem, which may also reduce the amount of bandwidth available in the device. Lower-level devices are therefore disproportionately affected by the reduction in memory and bandwidth.

[0034] In some aspects, described herein are systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for efficiently processing data and reducing resource burden in neural networks, such as pixel-to-pixel neural networks and any other neural networks that process large two-dimensional (2D) inputs (and / or three-dimensional (3D) inputs) into large or larger 2D outputs (and / or 3D outputs). The systems and techniques described herein may increase neural network efficiency while reducing the resource and processing burden and cost of the neural network. In some examples, the systems and techniques may provide architectures that implement stripe and raster-based data processing schemes that enable neural networks to process and / or stream higher amounts of data between neural network arrays and reduce the amount of data stored in memory during operation of the neural network. The systems and techniques described herein may reduce the amount of duplicated and / or redundant data processed by the neural network.

[0035] Examples of systems and techniques described herein for processing data are illustrated in Figures 1-9 and described below. Figure 1 illustrates an exemplary computing system 100. The computing system 100 may implement a neural network to process data. The neural network may implement architectures and data processing schemes to increase processing efficiency and / or reduce resource requirements, as described herein. The computing system 100 and / or the neural network may perform various tasks and operations, such as, for example, tasks and operations for extended reality, image / video processing and / or post-processing, data processing and / or post-processing, computer graphics, automation, machine vision, object modeling and object registration, multimedia rendering and / or composition, and / or any other data processing tasks, effects, and / or computations.

[0036] In some illustrative examples, the computing system 100 implements one or more inter-pixel neural networks and performs stripe and raster-based processing schemes and architectures, as further described herein. In some examples, the inter-pixel neural network may receive pixel inputs and generate pixel outputs. Although various examples of the present disclosure are described with respect to inter-pixel neural networks, the systems and techniques described herein may be implemented with and / or for other types and / or architectures of neural networks. For example, the systems and techniques described herein may be implemented for any neural network that processes large 2D data inputs into large or larger 2D data outputs, or any large 3D data inputs into large or larger 3D data outputs. The inter-pixel neural network examples herein are non-limiting examples provided for illustration and explanation.

[0037] In the example shown in FIG. 1, the computing system 100 includes one or more sensors 102, storage 104, memory 106, computing components 110, a processing engine 130, and a rendering engine 132. The computing system 100 may be part of a single computing device or multiple computing devices. In some examples, the computing system 100 may be part of an electronic device, such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a desktop computer, an XR device (e.g., head mounted display, etc.), a smart wearable device (e.g., smart watch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a gaming console, a video streaming device, a drone, an in-vehicle computer, a system on chip (SoC), an Internet of Things (IoT) device, or any other suitable electronic device.

[0038] In some implementations, one or more of the sensors 102, the storage 104, the memory 106, the computing component 110, the processing engine 130, and the rendering engine 132 may be part of the same computing device. For example, in some cases, one or more of the sensors 102, the storage 104, the memory 106, the computing component 110, the processing engine 130, and the rendering engine 132 may be integrated within a smartphone, a laptop, a tablet computer, a smart wearable device, a gaming system, a server, a SoC, and / or any other computing device. In other implementations, any of one or more of the sensors 102, the storage 104, the memory 106, the computing component 110, the processing engine 130, and / or the rendering engine 132 may be part of two or more separate computing devices.

[0039] The one or more sensors 102 may include one or more sensors, radar sensors, light detection and ranging (LIDAR) sensors, infrared (IR) sensors, gyroscopes, accelerometers, inertial measurement units (IMUs), audio sensors, and / or any other sensors. In some examples, the one or more sensors 102 may include one or more image sensors. The one or more image sensors may include image sensors and / or image capture devices capable of capturing images. For example, the one or more image sensors may include one or more sensors capable of capturing red-green-blue (RGB) images. In some examples, the image (or frame) may include a red-green-blue (RGB) image or frame having red, green, and blue color components per pixel, a luma, red-difference, blue-difference (YCbCr) image or frame having a luma component and two chroma (color) components (red difference and blue difference) per pixel, or any other suitable type of color or monochrome picture.

[0040] In some cases, the one or more image sensors may be any type of image sensor (or image capture device) and / or video sensor (or video capture device). For example, the one or more image sensors may include a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device included as part of an electronic device (e.g., a television, a computer, a camera, etc.). In some cases, the one or more image sensors may be part of a camera or computing device, such as a digital camera, a video camera, an IP camera, a smartphone, a smart television, a gaming system, a scanner, a multi-camera system, or other computing device. For example, the one or more image sensors may be part of a dual camera system or device, such as a smartphone, a camera, etc. The one or more image sensors may capture image data and / or video content (e.g., raw image and / or video data). The image data and / or video content may be processed by the computing component 110, the processing engine 130, and / or the rendering engine 132, as described herein.

[0041] The storage 104 may be any storage device for storing data, such as, for example, image, audio, file, or video data. Moreover, the storage 104 may also store data from any of the components of the computing system 100. For example, the storage 104 may store data or measurements from one or more sensors 102, the computing components 110 (e.g., parameters, inputs, outputs, generated images, calculation results, models, etc.), and / or data from any of the processing engines 130 and / or rendering engines 132 (e.g., output images, processing results, models, etc.). In some examples, the storage 104 may include a buffer for storing data (e.g., image data) for processing by the computing components 110. The memory 106 may include any memory device for storing data, such as a random access memory (RAM), a read-only memory (ROM), a cache, a memory buffer, etc.

[0042] In some implementations, the computing component 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, an image signal processor (ISP) 118, and / or a neural processing unit (NPU) 120. In some examples, the computing component 110 may include one or more memories, such as a cache and / or a memory buffer. The computing component 110 may perform various operations, such as data processing, computer vision, graphics rendering, extended reality (e.g., virtual reality, augmented reality, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, object recognition, feature recognition, face recognition, facial expression recognition, gaze recognition, age recognition, gender recognition, race recognition, tracking or pattern recognition, scene change recognition, etc.), machine learning, filtering, automation, computer graphics, and any of the various operations described herein. In some examples, the computing component 110 may implement a processing engine 130 and a rendering engine 132. In other examples, the computing component 110 may implement one or more other processing engines.

[0043] Operations for processing engine 130 and rendering engine 132 may be implemented by one or more of computing components 110. In one illustrative example, processing engine 130 (and related operations) may be implemented by CPU 112, DSP 116, ISP 118, and / or NPU 120, and rendering engine 132 (and related operations) may be implemented by GPU 114. In some cases, computing components 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof for performing any of the various operations described herein.

[0044] The computing component 110 may implement a processing engine 130 to perform various processing operations, such as image / video processing, XR rendering, computer vision, data processing, computer modeling, gaming, computer graphics, etc. The processing engine 130 may implement neural networks and processing architectures and / or schemes as further described herein. For example, the processing engine 130 may implement pixel-to-pixel neural networks and stripe and raster based processing schemes as described herein. In some examples, the processing engine 130 may be implemented by the NPU 120 to perform various tasks using stripe and raster based processing patterns as described herein.

[0045] Although computing system 100 is shown as including several components, one skilled in the art will appreciate that computing system 100 can include more or less components than those shown in Figure 1. For example, computing system 100 may also optionally include one or more additional memory devices (e.g., RAM, ROM, cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 1. Illustrative examples of computing devices and hardware components that may be implemented by computing system 100 are further described below with respect to Figure 9.

[0046] As mentioned above, some neural network architectures, such as pixel-to-pixel neural network architectures, use or require large memory buffers to process data. Memory buffer sizes may grow unsustainably as the size of the input data increases. Even large memory buffers may not be sufficient for some data processing cases. When memory buffers are oversubscribed and / or full, other resource requirements and / or utilization may increase substantially. For example, when memory buffers are oversubscribed, bandwidth, power, clock cycles, etc. also increase substantially. Adding more memory buffers and / or memory buffers with higher capacity may be expensive and may not adequately prevent and / or mitigate the increase in memory buffer oversubscription and / or other resource requirements and / or utilization. The cost of large memory buffers may also prevent the addition of other hardware resources, such as custom processors, to address such resource needs and / or limitations.

[0047] The systems and techniques described herein provide architectures and data processing schemes to address these and other challenges / limitations. The architectures and data processing schemes may be used to process any type of data, such as image data, video data, audio data, files, etc. For purposes of explanation and illustration, the architectures and data processing schemes are described with respect to processing image data.

[0048] In some examples, the systems and techniques described herein may provide stripe and raster based processing schemes. Stripe and raster based processing schemes may divide the input of a neural network into vertical stripes that the neural network processes according to a particular pattern as described below. Each layer of the neural network may process data in stripes along the depth and horizontal directions before moving vertically to process additional data. The additional data may be similarly processed along the depth and horizontal directions, again before moving vertically to process the new data.

[0049] 2 is a diagram illustrating an example input 200 for an inter-pixel neural network to implement a stripe and raster based processing scheme. The input 200 may include, for example, an image, an audio file, 2D data, 3D data, and / or any other type of data. In this example, the input 200 represents an image being processed by an inter-pixel neural network, such as an inter-pixel conventional neural network. The input 200 includes stripes 210A and stripes 210B, as further described herein.

[0050] In FIG. 2, the input 200 is shown in a left diagram 230A and a right diagram 230B showing stripes 210A and 210B at different locations within the input 200. The left diagram 230A shows stripes 210A at a left portion of the input 200 during a first processing time, and the right diagram 230B shows stripes 210B at a right portion of the input 200 during a second processing time. The input 200 is shown split in the diagrams 230A and 230B to show different stripes 210A and 210B within the input 200 at different times. Although the example of FIG. 2 shows two stripes, in other examples, the input 200 may include more or fewer stripes than shown in FIG. 2. In some examples, the number of stripes in the input 200 may be different or may depend on the neural network processing requirements and / or architecture.

[0051] Stripes 210A and 210B may each include several stripe blocks that extend vertically from the top of input 200 to the bottom of input 200. For example, FIG. 2 shows stripe block 212A at the top of stripe 210A and stripe block 212B at the bottom of stripe 210A, as well as stripe block 220A at the top of stripe 210B and stripe block 220B at the bottom of stripe 210B. These stripe blocks may each represent a row (or layer) of stripes 210A and 210B. Although not shown in FIG. 2 for simplicity, stripe 210A may include additional stripe blocks that extend from stripe block 212A to stripe block 212B such that stripe 210A includes a row (or layer) of stripe blocks from top to bottom. Similarly, although not shown in FIG. 2 for simplicity, stripe 210B may include additional stripe blocks extending from stripe block 220A to stripe block 220B, such that stripe 210B includes a row (or layer) of stripe blocks from top to bottom.

[0052] The input 200 and stripes 210A and 210B may include a horizontal dimension 202 (e.g., along the x-axis), a vertical dimension 204 (e.g., along the y-axis), and a depth dimension 206 (e.g., along the z-axis). Each layer of the neural network may process the data in the stripe along the depth dimension 206 and the horizontal dimension 202 before processing the data along the vertical dimension 204.

[0053] Stripe 210A may include overlap region 216, and stripe 210B may include overlap region 222. Overlap regions 216 and 222 may include an amount of overlap that is within the inputs processed by the neural network. As shown, the overlap within overlap regions 216 and 222 is in the horizontal dimension 202. Any overlap within the vertical dimension 204 may be stored in memory for use as needed by the neural network. In some cases, the overlap may result in a certain amount of extra and / or redundant calculations. However, the configurations and data processing schemes of stripes 210A and 210B described herein may avoid overlaps in other directions. Thus, the overlap within input 200 may be less than the amount of overlap in other data processing schemes and architectures, resulting in less extra and / or redundant calculations.

[0054] For example, the configurations and data processing schemes of stripes 210A and 210B described herein may avoid overlaps in at least one side along the horizontal dimension 202, at least one side along the vertical dimension 204, and one or both sides along the depth dimension 206 of a stripe block (e.g., stripe block 212A, stripe block 212B, stripe block 220A, stripe block 220B). This may significantly reduce the memory and processing requirements and / or burden of the neural network. Moreover, by including only a small amount of overlap along one side of the horizontal dimension 202 (for the stripe block) while processing and storing any amount of overlap that may exist along one side of the vertical dimension 204 (for the stripe block), the memory and processing requirements and / or burden of the neural network may be further reduced.

[0055] 3 illustrates an example data processing scheme 300 for a neural network 310. The data processing scheme 300 may include a stripe-raster data processing order or pattern. The neural network 310 may implement the data processing scheme 300 to process data in stripes 210A and 210B. The neural network 310 may include multiple layers (e.g., layers 312-318) for processing input data and generating output.

[0056] The neural network 310 may include any neural network that utilizes a large n-dimensional input and generates a large or larger n-dimensional output. For example, in some cases, the neural network 310 may include a pixel-to-pixel neural network. The pixel-to-pixel neural network may process input pixels from an image and generate output pixels for the image. In some examples, the image may be a large and / or high resolution image. The image may be, for example, a 2D image or a 3D image. In other cases, the neural network 310 may include a 2D input to 2D output neural network. For example, the neural network 310 may generate a 2D output from a 2D input. In some examples, the neural network 310 may process and / or transform each portion of the 2D data and generate a 2D data output for each portion of the 2D data. The 2D input and 2D output may include, for example, an image, video data, audio data, or any other 2D data.

[0057] In still other cases, the neural network 310 may include a 3D input to 3D output neural network. For example, the neural network 310 may generate a 3D output from a 3D input. In some examples, the neural network 310 may process and / or transform each portion of the 3D data and generate a 3D data output for each portion of the 3D data. The 3D input and 3D output may include, for example, images, video data, audio data, or any other 3D data.

[0058] In FIG. 3, stripe 210A and stripe 210B include block rows 306A-N for processing data in stripe block 302 according to data processing scheme 300. Stripe block 302 may include tiles 304 (or sub-blocks) of data that are first processed along the depth dimension 206 of stripe block 302 before shifting stripe block 302 along horizontal dimension 202 to process data in tiles 304 along the depth dimension 206 of stripe block 302 shifted in horizontal dimension 202. Tiles 304 may continue to be processed along the depth dimension 206 of stripe block 302, and stripe block 302 may continue to be processed along horizontal dimension 202 (e.g., across corresponding block rows). Once all of layers 312-318 have processed data across block rows, processing scheme 300 may move in vertical dimension 204 to similarly process data in the next row of the stripe.

[0059] Each layer of neural network 310 (e.g., each of layers 312-318) may process the data in stripe block 302 (and rows) according to the pattern described above before the next layer in neural network 310 processes the data in stripe block 302 (and rows) according to the same processing order / pattern. For example, each layer in neural network 310 may process the data in stripe block 302 in row 306A along the depth dimension 206 of stripe block 302, then along the horizontal dimension 202 of row 306A, before the next layer of the neural network processes the data in stripe block 302 in a similar pattern.

[0060] To illustrate, a layer 312 (e.g., layer 0) of neural network 310 may first process tiles 304 of stripe block 302 in row 306A of stripe 210A along the depth dimension 206 of stripe block 302. Once layer 312 has processed the data in stripe block 302 in row 306A, layer 312 may move along the horizontal dimension 202 and process data in adjacent stripe blocks in row 306A. Before layer 312 again moves along the horizontal dimension 202 of row 306A to the next adjacent stripe block, layer 312 may process tiles in adjacent stripe blocks along the depth dimension 206. Layer 312 may continue to process data in this pattern until layer 312 has finished processing the data in row 306A. Once layer 312 has completed processing the data in row 306A, the next layer in neural network 310, layer 314 (e.g., layer 1), may process the data in row 306A according to the same pattern. For example, layer 314 may process the data in the stripe blocks consecutively along the depth dimension and may process the blocks in row 306A consecutively along the horizontal dimension 202. Once layer 314 has completed processing the data in row 306A, the next layer in neural network 310, layer 316 (e.g., layer 2), may process the data in row 306A according to the same pattern. Once layer 314 has completed processing the data in row 306A, the next layer in neural network 310, layer 318 (e.g., layer N), may process the data in row 306A according to the same pattern.

[0061] In this manner, each layer in neural network 310 may successively process the data in row 306A according to the pattern described above before moving down the vertical dimension 204 of stripe 210A to similarly process (e.g., according to the same pattern) the data in the next row, row 306B. Once all layers in neural network 310 have completed processing the data in row 306B, neural network 310 may move down the vertical dimension 204 of stripe 210A, and each layer in neural network 310 may similarly process the data in the next row of stripe 210A. Neural network 310 may continue to move along the vertical dimension 204 until the data in the last row (e.g., row 306N) of stripe 210A has been processed by each layer.

[0062] Once stripe 210A is processed, neural network 310 may process the next stripe, stripe 210B. Neural network 310 may process stripe 210B according to the order / pattern described above with respect to stripe 210A until neural network 310 has completed processing all of the data in stripe 210B. As previously described, the input data may contain more or fewer stripes than shown in Figures 2 and 3. Thus, if the input data includes additional stripes, neural network 310 may proceed to the next stripe and process the data in the next stripe as previously described. Neural network 310 may continue to process any additional stripes until all stripes have been processed.

[0063] Each stripe (e.g., stripes 210A and 210B) may include any number of rows. Moreover, the size of the stripe blocks (e.g., stripe blocks 302) and tiles (e.g., tiles 304) is not limited to any particular size and may vary in different implementations. In some examples, the size of the stripe blocks and / or tiles may be based on preferences, neural network architecture, datapath architecture, neural network requirements, types and / or attributes of inputs, size and / or number of stripes, and / or any other factors. For example, in some cases, the size of the stripe blocks and / or tiles may be based on a multiply-accumulate (MAC) datapath architecture.

[0064] In some examples, the data processing scheme 300 may include an inner processing loop and an outer processing loop. The inner processing loop may process data in a stripe block according to the patterns previously described. In some examples, the inner processing loop may process horizontal blocks (or subtiles) in the stripe block sequentially or in parallel, process vertical blocks (or subtiles) in the stripe block sequentially or in parallel, process tile output depths sequentially or in parallel, and process input depths sequentially or in parallel. In one illustrative example, the horizontal blocks, vertical blocks, and tile output depths may be "unrolled" (e.g., processed and / or output in parallel) and the input depths may be "walked" (e.g., processed and / or output sequentially).

[0065] The outer loop may process the input data in a particular order, as previously described. For example, the outer loop may process the data in a depth direction, followed by a horizontal direction, followed by a neural network layer order, followed by a vertical direction, and followed by a stripe order. For example, the first layer of the neural network may first process the data in a stripe block along the depth dimension of the stripe block. The first layer of the neural network may then move horizontally to the next stripe block and similarly process the data in the next stripe block. The first layer of the neural network may continue to move horizontally and process the data in each stripe block along the horizontal direction until all blocks in the row have been processed. The second layer of the neural network may then execute the same pattern to process the stripe blocks in the row. Each subsequent layer in the neural network may similarly execute the same pattern until all layers have processed the stripe blocks in the row. Once all layers have processed the stripe blocks in the row, the first layer of the neural network may move vertically along the stripe to similarly process the stripe blocks in the next row. Each layer of the neural network can successively process stripe blocks in the next row, and the neural network can continue moving horizontally until all layers have processed stripe blocks in all rows. The neural network can then proceed to any subsequent stripes and process them successively according to the outer loop order described above.

[0066] 4 illustrates an example memory architecture 400 for storing stripe data for generating layer inputs. In this example, memory architecture 400 includes a line store memory 402 and a scratch memory 420. In some examples, line store memory 402 and scratch memory 420 may include one or more random access memories.

[0067] Scratch memory 420 may store outputs from layers (e.g., layers 312-318) of neural network 310. For example, buffer 422 in scratch memory 420 may store outputs from layer 0 (e.g., layer 312), and buffer 424 in scratch memory 420 may store outputs from layer 1 (e.g., layer 314). As additional layers of the neural network generate outputs, those outputs may be similarly stored in scratch memory 420. Outputs from a layer in the neural network may be used as inputs to the next layer in the neural network. In some examples, scratch memory 420 may include a buffer for each output of each layer of neural network 310. In some cases, the buffers (e.g., buffers 422 and 424) in the scratch memory may include ping-pong buffers.

[0068] The line store memory 402 may store portions of data from a layer in the neural network 310 (e.g., a line of data, a block of data, a window of data, a subset of data rows of a stripe block, etc.). The data stored in the line store memory 402 may include overlapping data of two or more layers of the neural network 310 (e.g., overlapping regions 216 and 222). In some examples, the line store memory 402 may include line stores 404-410. Each line store may correspond to a respective layer of the neural network 310, and the data in the line store may be used as part of the input for the layer associated with that line store. In this example, line store 404 may correspond to layer 0 (e.g., layer 312) of neural network 310, line store 406 may correspond to layer 1 (e.g., layer 314) of neural network 310, line store 408 may correspond to layer 2 (e.g., layer 316) of neural network 310, and line store 410 may correspond to layer N (e.g., layer 318) of neural network 310. Data stored in a line store associated with a layer of a neural network may include a subset of data from a previous layer of neural network 310 (e.g., a subset of one or more lines, one or more data blocks, an input window, or a stripe block of data rows).

[0069] For example, the line store 406 corresponding to layer 1 (e.g., layer 314) of the neural network 310 may contain a subset of output data from layer 0 (e.g., layer 312) of the neural network 310. In some cases, the size of the stored data may be based on the size of the convolution filter implemented by the neural network 310. For example, the convolution filter may cause a slight shift in the output data from each layer. Because a layer uses the output from the previous layer as part of its input, the shift in the output data may reduce the size of the input to that layer. The portion (e.g., overlap) of the data from the previous layer stored in the line store may thus be used as part of the input to the next layer to increase the size of the input to that next layer. To illustrate, the convolution filter implemented by a layer of the neural network 310 may cause a downscaling of the output data. Thus, the downscaled output from a layer may need to be augmented to build the input to the next layer.

[0070] As discussed above, the line store 406 corresponding to layer 1 (e.g., layer 314) of the neural network 310 may include a subset of output data from layer 0 (e.g., layer 312) of the neural network 310. Similarly, the line store 408 corresponding to layer 2 (e.g., layer 316) of the neural network 310 may include a subset of output data from layer 1 (e.g., layer 314) of the neural network 310, and the line store 410 corresponding to layer N (e.g., layer 318) of the neural network 310 may include a subset of output data from layer 2 (e.g., layer 316) of the neural network 310. In some examples, data stored in the line store 404 corresponding to layer 0 may include padding because layer 0 is the first layer and does not have data from previous layers. The padding may include, for example, edge duplication (e.g., repeating the first one or more input lines), value duplication, and / or any other padding technique and / or data.

[0071] In some examples, the overlap data from one layer of the neural network 310 may include a portion of the vertical overlap from the data processing scheme 300. For example, as previously described, the data processing scheme 300 may include overlap in the horizontal direction (e.g., overlap regions 216 and 222). In some cases, the horizontal overlap may be processed by each layer of the neural network 310 as each layer processes data in a row of the stripe (e.g., rows 306A, 306B, 306N). Additional overlap in the downward vertical direction may be stored in the line store memory 402 for use when processing the next row in the stripe. In some cases, this vertical overlap may include overlap data as described above with respect to the line stores 404-410. For example, the overlap data may include a portion of data vertically from a previous row of the stripe processed by the neural network 310.

[0072] The memory architecture 400 may prepare and process layer data using the outputs stored in the scratch memory 420 and the duplicate data in the line store memory 402. For example, in FIG. 4, the output 456 of layer 0 of the neural network 310 is stored in the buffer 422, and the duplicate data 454 from layer 0 of the neural network 310 is stored in the line store 406. To construct the input 452 for layer 1, the output 456 of layer 0 from the buffer 422 may be combined with the duplicate data 454 from the line store 406. The combination of the output 456 of layer 0 and the duplicate data 454 may result in the input 452 for layer 1.

[0073] As shown, memory architecture 400 can combine data from two memories (e.g., line store memory 402 and scratch memory 420) on a per-layer basis to form the input data for any given layer. In some cases, layers are processed consecutively without the need to fetch data from system memory.

[0074] In some examples, the line store memory 402 may store m lines of input data for each neural network layer. For example, the line store 404 may store m lines of input data for layer 0, the line store 406 may store m lines of input data for layer 1, the line store 408 may store m lines of input data for layer 2, and the line store 410 may store m lines of input data for layer N. In some cases, the m lines may depend on the kernel size of the associated neural network layer.

[0075] Each buffer in scratch memory 420 may store the output of a particular neural network layer. In some cases, the output of a particular neural network layer stored in a buffer may include n lines of data. For example, buffer 422 may store the output of layer 0, which includes n lines of data, and buffer 424 may store the output of layer 1, which includes n lines of data.

[0076] In some cases, n lines of data may be passed from layer to layer. In some examples, the n lines of data may be different in different examples and / or may depend on the architecture used. In some cases, increasing the number of lines n may allow for greater MAC parallelism. For example, if a MAC engine is 1024 MACs / cycle, the MAC engine may process several lines (e.g., n lines) simultaneously, several pixels (or other types of data) simultaneously, and several input and output depths simultaneously. In some cases, these parameters may be combined to form an inner loop tile size.

[0077] FIG. 5A illustrates an exemplary data processing architecture 500 for generating neural network layer inputs and outputs. The data processing architecture 500 includes a line store memory 402 and a scratch memory 420. As previously described, the scratch memory 420 may store outputs from layers (e.g., layers 312-318) of the neural network 310. For example, a buffer 422 in the scratch memory 420 may store outputs from layer 0 (e.g., layer 312), and a buffer 424 in the scratch memory 420 may store outputs from layer 1 (e.g., layer 314). As additional layers of the neural network generate outputs, those outputs may be similarly stored in the scratch memory 420. Outputs from a layer in the neural network may be used as inputs for the next layer in the neural network. In some examples, the scratch memory 420 may include as many buffers as there are buffers for each output of each layer of the neural network 310. In another example, scratch memory 420 may include a buffer for as few as two and a half block-row-layer outputs.

[0078] The line store memory 402 may store portions of data from layers in the neural network 310 (e.g., a line of data, a block of data, a window of data, a subset of data rows of a stripe block, etc.). The data stored in the line store memory 402 may include overlap data (e.g., overlap regions 216 and 222) of two or more layers of the neural network 310. In some examples, the overlap data from one layer of the neural network 310 may include portions of vertical overlap from the data processing scheme 300.

[0079] The data processing architecture 500 may prepare and process layer data using the output stored in the scratch memory 420 and the duplicate data in the line store memory 402. For example, in FIG. 5A , the output 540 of layer 0 of the neural network 310 is stored in the buffer 422, and the duplicate data 542 from a previous run of layer 1 of the neural network 310 is stored in the line store 406. To prepare and process data for layer 1, the vertical composer 530 may take the output 540 of layer 0 from the buffer 422 and the duplicate data 542 from the line store 406 and generate data 544 for layer 1 of the neural network 310. To generate data 544, the vertical composer 530 may combine the output 540 and the duplicate data 542. In some examples, the data 544 may include inputs constructed for layer 1.

[0080] Data 544 may be processed through MAC datapath 532 to generate layer 1 output 548. In some examples, output 548 may include blocks of data corresponding to stripe blocks within a row of a stripe. The blocks of data may include data from a horizontal position within the row of a stripe along the depth dimension of the stripe block. In other examples, output 548 may include layer 1 data output corresponding to a row of a stripe.

[0081] The layer 1 output 548 may then be stored in a buffer 424 on the scratch memory 420 for use by layer 2. In some examples, the MAC datapath 532 may include a MAC array and perform a matrix multiplication operation to generate the output 548. In some cases, the MAC datapath 532 may use a layer 1 convolution kernel to generate the output 548. For example, the convolution kernel may convolve the data to generate the output 548.

[0082] The line store update component 534 may obtain the data 544 generated by the vertical composer 530 to generate the duplicate data 546 for the next row of layer 1. The line store update component 534 may then store the duplicate data 546 in the line store 406 associated with layer 1. In some examples, the vertical composer 530 may determine and / or truncate a portion of the data 544 to generate the duplicate data 546. In some cases, the vertical composer 530 may determine and / or truncate a lower portion of the data 544 to generate the duplicate data 546. The data processing architecture 500 may then prepare and process data for layer 2 using the duplicate data 546 in the line store 408 and the output 548 in the buffer 424 from when layer 2 was previously executed.

[0083] In some examples, data processing architecture 500 may optionally include a horizontal composer 560 for adding horizontal duplicate data 550 to data 544 generated by vertical composer 530. For example, with reference to FIG. 5B , horizontal composer 560 may take data 544 from vertical composer 530 and add horizontal duplicate data 550 to data 544 to generate block inputs 552 for MAC array 562. Block inputs 552 may be used as MAC array inputs for MAC array 562. Block inputs 552 may include data 544 and horizontal duplicate data 550 added or appended to a horizontal edge of data 544.

[0084] In some examples, horizontal duplicate data 550 may pad data 544 with horizontal data (e.g., horizontal duplicate data 550) prior to data 544 being processed by MAC array 562. In some cases, by adding horizontal duplicate data 550 to data 544, data processing architecture 500 may reduce the overall bandwidth used by data processing architecture 500 and data processing scheme 300. In some examples, horizontal duplicate data 550 may include a subset of data from a previous horizontal location processed within the same row of a stripe as data 544.

[0085] The MAC array 562 may use the block inputs 552 to generate the outputs 548, which may be stored in the buffer 424 as previously described. In some examples, the MAC array 562 may perform a matrix multiplication operation to generate the outputs 548. In some cases, the horizontal composer 560 may implement an IM2COL operation on the data 544 and the horizontal duplicate data 550 to generate the block inputs 552.

[0086] 5B, data processing architecture 500 may include a write controller 570 for storing data in line store memory 402 and a read controller 572 for reading / retrieving data from line store memory 402. Similarly, data processing architecture 500 may include a write controller 574 for storing data in scratch memory 420 and a read controller 576 for reading / retrieving data from scratch memory 420.

[0087] 6 illustrates an example of a layer input 602 that is padded with a vertical offset and used to generate a layer output 610. In this example, the layer input 602 corresponds to a first pass of a particular neural network layer. The layer input 602 includes input lines 606 and padding 604. In some examples, the input lines 606 may include data retrieved from scratch memory 420, such as layer outputs stored in a buffer in scratch memory 420.

[0088] In some cases, padding 604 may be added before and / or on top of a first line 608 of the input lines 606. The padding 604 may provide a vertical line offset. In some examples, the vertical line offset may be used to increase the size of the layer input to accommodate the shifting and / or downscaling caused by the convolution filter as previously described. In some cases, the padding 604 may include duplicate data retrieved from the line store memory 402. The duplicate data may include one or more vertical lines from a previous stripe row processed by the layer.

[0089] In some cases, if the layer associated with the layer input 602 is layer 0 (e.g., layer 312) and no duplicate data from previous processing is available, the padding 604 may include repeated lines and / or values ​​used to pad the layer input 602. For example, the padding 604 may include one or more lines from the first line 608 and / or the input lines 606.

[0090] The layer inputs 602 may be used to generate the layer outputs 610. The layer outputs 610 may include output lines 612. In some examples, the output lines 612 may be stored in a buffer in the scratch memory 420 for use in constructing inputs for the next neural network layer.

[0091] As shown in FIG. 6, the layer input 602 is larger than the layer output 610. In some examples, convolutions in a neural network layer may cause a shift and / or reduction in the data. The shift and / or reduction in the data may cause the layer output 610 to be smaller than the layer input 602. Thus, when the layer output 610 is used to generate an input for a next layer, a similar padding (e.g., padding 604) of the layer output 610 may be used to increase the size of the input for the next layer. In some cases, overlapping data (e.g., one or more vertical lines from a previous layer and / or row) may be used for padding of the layer output 610.

[0092] 7 illustrates an example funnel processing scheme 700 for reducing redundant calculations when processing stripes. In this example, a central stripe 710 includes horizontal overlaps 712 on each side of the central stripe 710. The horizontal overlaps 712 are used to generate an output stripe 708 that corresponds to the central stripe 710. In this example, the output stripe 708 is generated using layers 0-N. A neural network (e.g., neural network 310) may process the stripe data layer-by-layer through layers 0-N.

[0093] Horizontal duplication 712 may increase the number of operations performed by a neural network (and / or layers of a neural network) and the strain on system resources, such as bandwidth, memory, power, and / or computation, that may increase when each additional layer is implemented by a neural network. In some examples, the more layers implemented by a neural network, the greater the strain on resources and the number of redundant operations performed by the layers of the neural network.

[0094] The funnel processing scheme 700 may reduce resource burden and redundant computation by progressively reducing the active stripe width as the neural network progresses layer by layer. In some examples, the funnel processing scheme 700 may also reduce the number of MACs per layer.

[0095] The funnel processing scheme 700 may first generate an input stripe 702 for layer 0. As shown in FIG. 7, the input range of the input stripe 702 for layer 0 is greater than the input ranges for layers 1, 2, and N. For example, when going from layer 0 to layer 1, the funnel processing scheme 700 may reduce the input range of an intermediate stripe 704 for layer 1 compared to the input range of the input stripe 702 for layer 0. Similarly, the funnel processing scheme 700 may reduce the input range of an intermediate stripe 706 for layer 2 compared to the input range of the intermediate stripe 704 for layer 1. The funnel processing scheme 700 may further reduce the input range for an output stripe 708 at layer N compared to the input range of the intermediate stripe 706 for layer 2.

[0096] In this manner, the funnel processing scheme 700 may progressively reduce the input range for each layer of the neural network, as well as the redundant calculations used to generate the output stripes 708. The reduction in calculations and processing data may reduce the strain on resources, such as memory, computation, power, and bandwidth.

[0097] 8 is a flow chart illustrating an example process 800 for processing data in a neural network, according to some examples of the present disclosure. At block 802, the process 800 may include obtaining input data (e.g., input 200) including stripes (e.g., stripe 210A, stripe 210B) that logically segment the input data. Each stripe of the input data includes at least one row of data, such as stripe 210A shown in FIG. 2. In some examples, the process 800 may include logically segmenting the input data into stripes. In some cases, each stripe may include a respective portion of the input data.

[0098] At block 804, process 800 may include processing a row of data (e.g., row 306A) in a first stripe (e.g., stripe 210A) of input data by each layer (e.g., layer 312, layer 314, layer 316, layer 318) of a neural network (e.g., neural network 310). In some examples, the rows of data may be processed successively in a layer-by-layer sequence, horizontally (e.g., horizontal dimension 202), with each preceding layer of the neural network processing a particular row of data (e.g., a row of data in the first stripe, such as row 306A in stripe 210A) before each subsequent layer of the neural network.

[0099] In some cases, sequentially processing a row of data horizontally may include sequentially processing multiple blocks of data (e.g., stripe blocks 302) within the row by each layer of the neural network. In some cases, each layer of the neural network may process each preceding block of data along a depth direction (e.g., depth dimension 206) before processing a subsequent block along a horizontal direction (e.g., horizontal dimension 202).

[0100] In some cases, the neural network may include a convolutional network. In some examples, the neural network may include a pixel-to-pixel neural network, and the input data may include pixels associated with an image. In other cases, the neural network may include a neural network that processes 2D data input into 2D data output and / or 3D data input into 3D data output.

[0101] In some examples, processing the rows of data according to the layer-by-layer sequence may include processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network. In some examples, the output may correspond to the rows of data.

[0102] At block 806, process 800 may include processing subsequent rows of data in the first stripe (e.g., rows 306B, 306N) on a row-by-row basis by each layer of the neural network after processing the rows of data in the first stripe. In some examples, each subsequent row of data may be processed successively in a horizontal, layer-by-layer sequence (e.g., each preceding layer of the neural network processes a subsequent row before each subsequent layer of the neural network).

[0103] In some cases, processing the subsequent rows may include sequentially processing, by each layer of the neural network, each of the blocks of data (e.g., stripe blocks 302) in each subsequent row. In some cases, each layer of the neural network may process a preceding block of data in a subsequent row in the first stripe depth-wise before processing a subsequent block of data in the subsequent row.

[0104] In some cases, processing subsequent rows of data according to a layer-by-layer sequence may include processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network. In some examples, the output may correspond to a subsequent row of data.

[0105] At block 808, the process 800 may include generating, by the neural network, a first output stripe based on processing the row of data and the subsequent row of data.

[0106] At block 810, the process 800 may include processing rows of data in a second stripe (e.g., stripe 210B) of the input data after processing the first stripe through each layer of the neural network. In some examples, the rows of data in the second stripe may be processed on a row-by-row basis. In some examples, each row of data in the second stripe may be processed according to a horizontal, layer-by-layer sequence.

[0107] At block 812, process 800 may include generating a second output stripe based on the processing of the second stripe. In some examples, the neural network may generate an output for the input data based on the first output stripe and / or the second output stripe.

[0108] In some cases, process 800 may include storing an output (e.g., output 548) generated by a previous layer of the neural network for one or more blocks (e.g., stripe block 302) in the row of data in a first memory (e.g., scratch memory 420) and storing one or more lines of data from one or more blocks in the row of data in a second memory (e.g., line store memory 402) associated with a particular layer of the neural network. In some examples, the one or more lines of data may include portions of a data input of a particular layer of the neural network for a previous stripe-block-row (e.g., duplicate data 542, duplicate data 546).

[0109] In some examples, process 800 may include generating an input (e.g., input 452, block input 552) for a particular layer of the neural network based on a combination of one or more lines of data that include an output generated by a previous layer and a portion of the data input for this same layer for a previous stripe-row. Process 800 may include generating additional outputs by a particular layer of the neural network based on the input for the previous layer.

[0110] In some examples, process 800 may include determining and / or pruning portions of the input for a particular layer (e.g., overlap data 542, overlap data 546). Process 800 may include storing the portions of the input in a third memory (e.g., line store memory 402) associated with the particular layer of the neural network.

[0111] In some cases, process 800 may include generating an additional input for a subsequent layer of the neural network based on a combination of the additional output of a particular layer and a portion of the input from a previous block-row of the subsequent layer. Process 800 may include generating a second additional output by the subsequent layer of the neural network based on the additional input for the subsequent layer.

[0112] In some cases, process 800 may include storing the second additional output in a fourth memory (e.g., scratch memory 420). In some examples, the second memory and the third memory may include line stores (e.g., line store 404, line store 406, line store 408, line store 410) in a line store memory (e.g., line store memory 402), and the first memory and the fourth memory may include buffers (e.g., buffer 422, buffer 424) in a scratch memory (e.g., scratch memory 420).

[0113] In some cases, process 800 may include storing outputs generated by a previous layer of the neural network for one or more blocks in each subsequent row of data in a first memory (e.g., scratch memory 420) associated with the particular layer of the neural network. Process 800 may include storing one or more lines of data from one or more blocks in each subsequent row of data in a second memory (e.g., line store memory 402). In some examples, the one or more lines of data may include one or more portions of one or more data inputs of a particular layer of the neural network.

[0114] In some cases, process 800 may include generating an input for a particular layer of the neural network based on a combination of one or more lines of data that include one or more portions of the output generated by a previous layer and one or more data inputs of the particular layer for the previous stripe-block-row. Process 800 may include generating additional outputs by the particular layer of the neural network based on the output from the previous layer and a subset of the inputs from the particular layer for the previous stripe-block-row.

[0115] In some cases, process 800 may include determining and / or pruning a portion of the input for a particular layer. Process 800 may include storing the portion of the input in a third memory (e.g., line store memory 402) associated with a subsequent layer of the neural network. Process 800 may further include storing additional outputs in a fourth memory (e.g., scratch memory 420). In some examples, the second memory and the fourth memory may include memory buffers (e.g., buffer 422, buffer 424).

[0116] In some examples, the processes described herein (e.g., process 800, and / or other processes described herein) may be performed by a computing device or apparatus. In one example, process 800 may be performed by computing system 100 of FIG. 1. In another example, process 800 may be performed by a computing system having a computing device architecture 900 shown in FIG. 9. For example, a computing device with computing device architecture 900 shown in FIG. 9 may implement the operations of FIG. 8 and / or the components and / or operations described herein with respect to any of FIGS. 1-8.

[0117] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or autonomous vehicle computing device, a robotic device, a laptop computer, a smart television, a camera, and / or any other computing device with resource capabilities to perform the processes described herein, including process 800 and / or any other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0118] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0119] Process 800 is illustrated as a logical flow diagram, whose operations represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0120] Additionally, process 800, and / or other processes described herein, may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0121] 9 illustrates an exemplary computing device architecture 900 of an exemplary computing device that may implement various techniques described herein. For example, the computing device architecture 900 may implement at least some portions of the computing system 100 illustrated in FIG. 1. The components of the computing device architecture 900 are shown as being in electrical communication with each other using connections 905, such as a bus. The exemplary computing device architecture 900 includes a processing unit (CPU or processor) 910 and computing device connections 905 that couple various computing device components, including a computing device memory 915, such as a read only memory (ROM) 920 and a random access memory (RAM) 925, to the processor 910.

[0122] The computing device architecture 900 may include a cache of high-speed memory directly connected to, near, or integrated as part of the processor 910. The computing device architecture 900 may copy data from the memory 915 and / or storage device 930 to the cache 912 for fast access by the processor 910. In this manner, the cache may provide improved performance that eliminates delays for the processor 910 while waiting for data. These and other modules may control or be configured to control the processor 910 to perform various actions. Other computing device memories 915 may also be available. The memory 915 may include multiple different types of memory with different performance characteristics. The processor 910 may include any general-purpose processor, as well as hardware or software services (e.g., service 1 932, service 2 934, and service 3 936) stored in the storage device 930 and configured to control the processor 910, as well as special-purpose processors where software instructions are built into the processor design. The processor 910 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0123] To enable user interaction with the computing device architecture 900, the input devices 945 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output devices 935 may be one or more of several output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 900. The communication interface 940 can generally govern and manage user input and computing device output. Since there are no constraints acting on any particular hardware configuration, this basic feature may be easily replaced by improved hardware or firmware configurations as they are developed.

[0124] The storage device 930 is a non-volatile memory and may be a hard disk or other type of computer readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 925, a read only memory (ROM) 920, and hybrids thereof. The storage device 930 may include services 932, 934, 936 for controlling the processor 910. Other hardware or software modules are contemplated. The storage device 930 may be connected to the computing device connection 905. In one aspect, a hardware module that performs a particular function may include software components stored in a computer readable medium that connects with necessary hardware components, such as the processor 910, the connection 905, the output device 935, etc., to perform the function.

[0125] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or transport instructions and / or data. Computer-readable media may include non-transitory media, which do not include carrier waves and / or transitory electronic signals on which data may be stored and which propagate wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may store code and / or machine-executable instructions, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0126] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0127] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the present embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks comprising devices, device components, and steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, or other components may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0128] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0129] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions, or that otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the examples described include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network-attached storage devices, etc.

[0130] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board, in different chips, or on different processes executing in a single device, as further examples.

[0131] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0132] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the applications described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0133] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.

[0134] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0135] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communications interface).

[0136] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include items not recited in the set of A and B.

[0137] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0138] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0139] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0140] Illustrative examples of the present disclosure include the following:

[0141] Aspect 1: An apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors acquiring input data comprising stripes that logically segment the input data, each stripe of the input data including at least one row of data; processing rows of data in a first stripe of the input data by each layer of a neural network, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network; and processing, after processing the rows of data in the first stripe, each layer of the neural network. and generating a first output stripe based on the processing of the row of data and the subsequent row of data by the neural network; and after processing the first stripe, processing rows of data in a second stripe of input data on a row-by-row basis by each layer of the neural network, where each row of data in the second stripe is processed in the horizontal direction in accordance with the layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0142] Aspect 2: The apparatus of aspect 1, wherein to sequentially process rows of data in the first stripe horizontally, the one or more processors are configured to sequentially process multiple blocks of data in the rows with each layer of a neural network, each layer of the neural network processing each previous block of data depthwise before processing a subsequent block horizontally.

[0143] Aspect 3: The apparatus of aspect 2, wherein to process subsequent rows, the one or more processors are configured to sequentially process, through each layer of the neural network, a respective plurality of blocks of data in each subsequent row, where each layer of the neural network processes a preceding block of data in the subsequent row depthwise before processing a subsequent block of data in the subsequent row.

[0144] Example 4: The apparatus of any one of examples 1 to 3, wherein the neural network comprises a pixel-to-pixel neural network and the input data comprises pixels associated with an image.

[0145] Aspect 5: The apparatus of any of aspects 1 to 4, wherein, to obtain the input data, the one or more processors are configured to: logically segment the input data into stripes, each stripe including a respective portion of the input data.

[0146] Aspect 6: The apparatus of any of aspects 1-5, wherein the one or more processors are configured to: store in a first memory output generated by a previous layer of the neural network for one or more blocks in a row of data in the first stripe; and store in a second memory associated with a particular layer of the neural network one or more lines of data from the one or more blocks in the row of data, wherein the one or more lines of data include a portion of a data input of the particular layer of the neural network for the previous stripe-block-row.

[0147] Aspect 7: The apparatus of aspect 6, wherein the one or more processors are configured to generate an input for a particular layer of the neural network based on a combination of an output generated by a previous layer and one or more lines of data that include a portion of the particular layer's data input for a previous stripe-block-row, and to generate an additional output by the particular layer of the neural network based on the output for the previous layer.

[0148] Example 8: The apparatus of example 7, wherein the one or more processors are configured to determine a portion of an input for a particular layer and store the portion of the input in a third memory associated with the particular layer of the neural network.

[0149] Aspect 9: The apparatus of aspect 8, wherein the one or more processors are configured to generate an additional input for a subsequent layer of the neural network based on a combination of the additional output of a particular layer and a portion of the input for the subsequent layer from a previous stripe-block-row of the subsequent layer, and generate, by the subsequent layer of the neural network, a second additional output based on the additional input for the subsequent layer.

[0150] Aspect 10: The apparatus of aspect 9, wherein the one or more processors are configured to store the second additional output in a fourth memory, wherein the second memory and the third memory comprise a line store in a line store memory, and the first memory and the fourth memory comprise a buffer in a scratch memory.

[0151] Aspect 11: The apparatus of any of aspects 1-10, wherein the one or more processors are configured to: store in a first memory associated with a particular layer of the neural network outputs generated by a previous layer of the neural network for one or more blocks in each subsequent row of data; and store in a second memory one or more lines of data from the one or more blocks in each subsequent row of data, where the one or more lines of data include one or more portions of one or more data inputs of the particular layer of the neural network.

[0152] Aspect 12: The apparatus of aspect 11, wherein the one or more processors are configured to generate an input for a particular layer of the neural network based on a combination of one or more lines of data including one or more portions of outputs generated by a previous layer and one or more data inputs of the particular layer for the previous stripe-block-row, and to generate an additional output by the particular layer of the neural network based on the outputs from the previous layer and a subset of the inputs from the particular layer for the previous stripe-block-row.

[0153] Aspect 13: The apparatus of aspect 12, wherein the one or more processors are configured to determine a portion of an input for a particular layer, store the portion of the input in a third memory associated with a subsequent layer of the neural network, and store an additional output in a fourth memory, wherein the second memory and the fourth memory comprise memory buffers.

[0154] Aspect 14: The apparatus of any of aspects 1 to 13, wherein the one or more processors are configured to process, by each subsequent layer of the neural network, an output of a preceding layer of the neural network, where the output corresponds to the row of data, to process the row of data.

[0155] Aspect 15: The apparatus of any of aspects 1 to 14, wherein the one or more processors are configured to process, by each subsequent layer of the neural network, an output of a previous layer of the neural network, where the output corresponds to a subsequent row of data in the first stripe, to process subsequent rows of data.

[0156] Example 16: The apparatus of any one of examples 1 to 15, wherein the apparatus comprises a camera device.

[0157]

[0031] Example 17: The apparatus of any of examples 1 to 16, wherein the apparatus comprises a mobile device.

[0158] Aspect 18: A method, comprising: obtaining input data with stripes that logically segment the input data, each stripe of the input data including at least one row of data; processing, by each layer of a neural network, a row of data in a first stripe of the input data, the rows of data being processed horizontally successively according to a layer-by-layer sequence, with each preceding layer of the neural network processing a particular row of data before each subsequent layer of the neural network; and processing, by each layer of the neural network after processing the row of data in the first stripe, subsequent rows of data in the first stripe. a step of processing rows on a row-by-row basis, where each subsequent row of data is processed successively in a horizontal, layer-by-layer sequence; generating, by a neural network, a first output stripe based on the processing of the row of data and the subsequent row of data; after processing the first stripe, processing, by each layer of the neural network, rows of data in a second stripe of input data on a row-by-row basis, where each row of data in the second stripe is processed in a horizontal, layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0159] Aspect 19: The method of aspect 18, wherein the step of horizontally sequentially processing rows of data in the first stripe includes a step of sequentially processing multiple blocks of data in the rows by each layer of the neural network, each layer of the neural network processing each preceding block of data depthwise before processing a subsequent block horizontally.

[0160] Aspect 20: The method of aspect 19, wherein the step of processing subsequent rows of data includes the step of sequentially processing, by each layer of the neural network, a respective plurality of blocks of data in each subsequent row, wherein each layer of the neural network processes, depth-wise, a preceding block of data in the subsequent row before processing a subsequent block of data in the subsequent row.

[0161] Example 21: The method of any of examples 18 to 20, wherein the neural network comprises a pixel-to-pixel neural network and the input data comprises pixels associated with an image.

[0162] Aspect 22: The method of any of aspects 18 to 21, wherein obtaining the input data comprises logically segmenting the input data into stripes, each stripe including a respective portion of the input data.

[0163] Aspect 23: The method of any of aspects 18 to 22, further comprising: storing in a first memory an output generated by a previous layer of the neural network for one or more blocks in a row of data in the first stripe; and storing in a second memory associated with a particular layer of the neural network one or more lines of data from one or more blocks in the row of data, the one or more lines of data comprising a portion of a data input of the particular layer of the neural network for the previous stripe-block-row.

[0164] Aspect 24: The method of aspect 23, further comprising: generating an input for a particular layer of the neural network based on a combination of an output generated by a previous layer and one or more lines of data comprising a portion of the particular layer's data input for the previous stripe-block-row; and generating an additional output by the particular layer of the neural network based on the output for the previous layer.

[0165] Example 25: The method of example 24, further comprising determining a portion of the input for a particular layer and storing the portion of the input in a third memory associated with the particular layer of the neural network.

[0166] Aspect 26: The method of aspect 25, further comprising: generating an additional input for a subsequent layer of the neural network based on a combination of the additional output of the particular layer and a portion of the input for the subsequent layer from a previous stripe-block-row of the subsequent layer; and generating a second additional output by the subsequent layer of the neural network based on the additional input for the subsequent layer.

[0167] Aspect 27: The method of aspect 26, further comprising the step of storing the second additional output in a fourth memory, wherein the second memory and the third memory comprise a line store in a line store memory, and the first memory and the fourth memory comprise a buffer in a scratch memory.

[0168] Aspect 28: The method of any of aspects 18 to 27, further comprising: storing in a first memory associated with the particular layer of the neural network an output generated by a previous layer of the neural network for one or more blocks in each subsequent row of data; and storing in a second memory one or more lines of data from the one or more blocks in each subsequent row of data, the one or more lines of data including one or more portions of one or more data inputs of the particular layer of the neural network.

[0169] Aspect 29: The method of aspect 28, further comprising: generating an input for a particular layer of the neural network based on a combination of the output generated by a previous layer and one or more lines of data including one or more portions of the one or more data inputs of the particular layer for the previous stripe-block-row; and generating an additional output by the particular layer of the neural network based on the output from the previous layer and a subset of the inputs from the particular layer for the previous stripe-block-row.

[0170] Aspect 30: The method of aspect 29, further comprising the steps of determining a portion of an input for a particular layer, storing the portion of the input in a third memory associated with a subsequent layer of the neural network, and storing an additional output in a fourth memory, wherein the second memory and the fourth memory comprise memory buffers.

[0171] Aspect 31: The method of any of aspects 18 to 30, wherein the step of processing the row of data includes a step of processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network, where the output corresponds to the row of data.

[0172] Aspect 32: The method of any of aspects 18 to 31, wherein the step of processing subsequent rows of data includes a step of processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network, where the output corresponds to a subsequent row of data in the first stripe.

[0173] Aspect 33: A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 18 to 32.

[0174] Embodiment 34: An apparatus comprising means for carrying out the method according to any of embodiments 18 to 32. [Explanation of symbols]

[0175] 100 Computing Systems 102 Sensors 104 Storage 106 Memory 110 Computing Components 112 Central Processing Unit (CPU) 114 Graphics Processing Unit (GPU) 116 Digital Signal Processor (DSP) 118 Image Signal Processor (ISP) 120 Neural Processing Units (NPU) 130 Processing Engine 132 Rendering Engine 200 Input 202 horizontal dimension 204 Vertical dimension 206 Depth Dimension 210A Stripe 210B Stripe 212A Stripe Block 212B Stripe Block 216 Overlapping area 220A Stripe Block 220B Stripe Block 222 Overlapping area 230A Left figure, Fig. 230B Right figure, Fig. 300 Data processing method, processing method 302 Stripe Block 304 tiles Line 306A Line 306B Row 306N 310 Neural Networks 312~318 Layer 400 Memory Architecture 402 Line Store Memory 404~410 Line Store 420 Scratch Memory 422 Buffer 424 Buffer 452 Input 454 Duplicate Data 456 Output 500 Data Processing Architecture 530 Vertical Composer 532 MAC Data Path 534 Line Store Update Component 540 Output 542 duplicate data 544 Data 545 Duplicate Data 546 Duplicate Data 548 Output 550 Horizontally overlapping data 552 Block Input 560 Horizontal Composer 562 MAC Array 570 Write Controller 572 Read Controller 574 Write Controller 576 Read Controller 602 Layer Input 604 Padding 606 Input Line 608 First Line 610 Line Layer Output, Layer Output 612 Output Line 700 Funnel-type treatment method 702 input stripes 704 Middle Stripe 706 Middle Stripe 708 Output Stripe 710 Center Stripe 712 horizontal overlap 800 processes 900 Computing Device Architecture 905 Connection, Computing Device Connection 910 Processing Unit (CPU or Processor) 912 Cache 915 Computing device memory, memory 920 Read-Only Memory (ROM) 925 Random Access Memory (RAM) 930 Storage Device 932 Service 1, Service 934 Service 2, Service 935 Output Device 936 Service 3, Service 940 Communication Interface 945 Input Devices

Claims

1. An apparatus comprising: Memory, one or more processors coupled to the memory; wherein the one or more processors: obtaining input data comprising stripes that logically segment the input data, each stripe of the input data including at least one row of data; processing rows of data in a first stripe of the input data by each layer of a neural network, the rows of data being processed horizontally successively according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network; processing, after processing the row of data in the first stripe, subsequent rows of data in the first stripe on a row-by-row basis by each layer of the neural network, where each subsequent row of data is processed successively in the horizontal direction according to the layer-by-layer sequence; generating, by the neural network, a first output stripe based on the processing of the row of data and the subsequent row of data; processing, by each layer of the neural network after processing the first stripe, rows of data in a second stripe of the input data on a row-by-row basis, where each row of data in the second stripe is processed in the horizontal direction in accordance with the layer-by-layer sequence; generating a second output stripe based on said processing of the second stripe; An apparatus configured to:

2. to process the rows of data in the first stripe sequentially in the horizontal direction, the one or more processors: Successively processing a number of blocks of data in the row with each layer of the neural network, each layer of the neural network processing each preceding block of data along the depth direction before processing a subsequent block along the horizontal direction. configured to: To process the subsequent row, the one or more processors: Successively processing a respective plurality of blocks of data in each subsequent row by each layer of the neural network, each layer of the neural network processing a preceding block of data in the subsequent row along the depth direction before processing a subsequent block of data in the subsequent row. The apparatus of claim 1 , configured to:

3. the neural network includes an inter-pixel neural network; The apparatus of claim 1 , wherein the input data comprises pixels associated with an image.

4. To obtain the input data, the one or more processors: logically segmenting the input data into stripes, each stripe including a respective portion of the input data; The apparatus of claim 1 , configured to:

5. the one or more processors: storing in a first memory outputs generated by a previous layer of the neural network for one or more blocks in the row of data in the first stripe; storing one or more lines of data from the one or more blocks in the row of data in a second memory associated with a particular layer of the neural network, the one or more lines of data including portions of a data input of the particular layer of the neural network for a previous stripe-block-row; The apparatus of claim 1 , configured to:

6. the one or more processors: generating inputs for the particular layer of the neural network based on a combination of the outputs generated by the previous layer and the one or more lines of data comprising the portion of the data input of the particular layer for the previous stripe-block-row; generating additional outputs by the particular layer of the neural network based on the outputs for the previous layer; and The apparatus of claim 5 , configured to:

7. the one or more processors: determining a portion of the input for the particular layer; and storing the portion of the input in a third memory associated with the particular layer of the neural network; The apparatus of claim 6 , configured to:

8. the one or more processors: generating additional inputs for the subsequent layer of the neural network based on a combination of the additional outputs of the particular layer and the portion of the inputs for the subsequent layer from a previous stripe-block-row of the subsequent layer; generating, by the subsequent layer of the neural network, a second additional output based on the additional input to the subsequent layer; [0023] the one or more processors: storing the second additional output in a fourth memory, wherein the second memory and the third memory comprise a line store in a line store memory, and the first memory and the fourth memory comprise a buffer in a scratch memory. The apparatus of claim 7 , configured to:

9. the one or more processors: storing, in a first memory associated with a particular layer of the neural network, outputs generated by a previous layer of the neural network for one or more blocks in each subsequent row of data; storing in a second memory one or more lines of data from the one or more blocks in each subsequent row of data, the one or more lines of data including one or more portions of one or more data inputs of the particular layer of the neural network; The apparatus of claim 1 , configured to:

10. the one or more processors: generating inputs for the particular layer of the neural network based on a combination of the outputs generated by the previous layer and the one or more lines of data comprising the one or more portions of the one or more data inputs of the particular layer for a previous stripe-block-row; generating, by the particular layer of the neural network, an additional output based on the output from the previous layer and a subset of the inputs from the particular layer for the previous stripe-block-row; [0023] the one or more processors: determining a portion of the input for the particular layer; and storing the portion of the input in a third memory associated with a subsequent layer of the neural network; storing the additional output in a fourth memory, the second memory and the fourth memory comprising a memory buffer; The apparatus of claim 9 , configured to:

11. To process the rows of data, the one or more processors: processing, by each subsequent layer of the neural network, an output of a preceding layer of the neural network, the output corresponding to the row of data; The apparatus of claim 1 , configured to:

12. To process the subsequent row of data, the one or more processors: processing, by each subsequent layer of the neural network, an output of a previous layer of the neural network, the output corresponding to a subsequent row of data in the first stripe; The apparatus of claim 1 , configured to:

13. The apparatus of claim 1 , wherein the apparatus comprises a camera device and / or a mobile device.

14. 1. A method comprising: obtaining input data with stripes that logically segment the input data, each stripe of the input data comprising at least one row of data; processing rows of data in a first stripe of the input data by each layer of a neural network, the rows of data being processed horizontally successively according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network; after processing the rows of data in the first stripe, processing subsequent rows of data in the first stripe on a row-by-row basis by each layer of the neural network, each subsequent row of data being processed successively in the horizontal direction according to the layer-by-layer sequence; generating, by the neural network, a first output stripe based on the processing of the row of data and the subsequent row of data; after processing the first stripe, processing rows of data in a second stripe of the input data on a row-by-row basis by each layer of the neural network, where each row of data in the second stripe is processed in the horizontal direction in accordance with the layer-by-layer sequence; generating a second output stripe based on said processing of the second stripe; A method comprising:

15. 15. A non-transitory computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform the method of claim 14.

Images