Data processing in inter-pixel neural networks

Abstract

Techniques are provided for processing data in a neural network. An exemplary method may include processing rows of a first stripe of a data input by each layer of the neural network, where the rows are processed horizontally and consecutively according to a layer-wise sequence, processing subsequent rows in the first stripe by each layer after processing the rows on a row-by-row basis, where each subsequent row is processed horizontally and consecutively according to a layer-wise sequence, generating an output stripe based on the processing of the rows and subsequent processing, processing a second stripe of the data input by each layer, where each row in the second stripe is processed horizontally and consecutively according to a layer-wise sequence and the rows in the second stripe are processed on a row-by-row basis, and generating another output stripe based on the processing of the second stripe.

Description

Technical Field

[0001] The present disclosure generally relates to neural networks. For example, aspects of the present disclosure relate to techniques and systems for processing data in neural networks.

Background Art

[0002] Various devices and systems may implement processing engines and networks to process various types of data and generate certain outputs and / or effects. For example, some devices may implement a neural network to process images and / or videos and generate an output for consumption (e.g., for display on the device, for use by an application, etc.). Neural networks can effectively execute various complex tasks with accurate results. The outputs and functions of neural networks can also be used in various applications such as, among other things, computer graphics, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition, data processing (e.g., image processing, video processing, etc.), data post-processing, visual rendering, television, and the like.

[0003] Neural networks often process large amounts of data, such as video, images, and / or audio data, to perform a specific task and produce a particular output. The amount of data processed by neural networks and the processing demands of the neural networks can place a considerable burden on device resources, such as memory, power resources, and computing resources. Furthermore, some neural network architectures may be more resource-intensive than others and / or may require or utilize additional resources for their operation. This further increases the resource and / or processing costs of the neural network, reduces its efficiency, and / or, in some cases, increases its operational burden, resource burden, and requirements. [Overview of the project] [Means for solving the problem]

[0004] Several examples describe techniques and systems for increasing neural network efficiency and reducing the resources, processing burden, and cost of neural networks. According to at least one exemplary example, a method for processing data in an inter-pixel neural network is provided. In some examples, the method includes the steps of: acquiring input data having stripes that logically segment the input data, such that each stripe of the input data contains at least one row of data; processing rows of data in a first stripe of the input data by each layer of the neural network, such that the rows of data are processed sequentially horizontally according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each succeeding layer of the neural network; and processing rows of data in the first stripe by each layer of the neural network after processing the rows of data in the first stripe. The process may include: processing subsequent rows of data on a row-by-row basis, wherein each subsequent row of data is processed sequentially horizontally according to a layer-by-layer sequence; generating a first output stripe by a neural network based on the processing of the rows of data and subsequent rows of data; processing the rows of data in a second stripe of input data on a row-by-row basis by each layer of the neural network after processing the first stripe, wherein each row of data in the second stripe is processed horizontally according to a layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0005] According to at least one exemplary example, a non-temporal computer-readable medium for processing data in an inter-pixel neural network is provided. In some embodiments, the non-temporal computer-readable medium may include instructions, which, when executed by one or more processors, instruct one or more processors to acquire input data having stripes that logically segment the input data, wherein each stripe of the input data contains at least one row of data; and each layer of the neural network processes the row of data in a first stripe of the input data, wherein the row of data is processed sequentially horizontally according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network; and after processing the row of data in the first stripe, Each layer of the neural network processes subsequent rows of data in the first stripe on a row-by-row basis, with each subsequent row of data being processed sequentially horizontally according to a layer-by-layer sequence; the neural network generates a first output stripe based on the processing of the data rows and subsequent rows of data; and after processing the first stripe, each layer of the neural network processes rows of data in the second stripe of input data on a row-by-row basis, with each row of data in the second stripe being processed horizontally according to a layer-by-layer sequence; and a second output stripe is generated based on the processing of the second stripe.

[0006] According to at least one exemplary example, an apparatus for processing data in an inter-pixel neural network is provided. In some embodiments, the apparatus may include a memory storing computer-readable instructions and one or more processors, the one or more processors comprising: acquiring input data having stripes that logically segment the input data, wherein each stripe of the input data contains at least one row of data; and processing the rows of data in a first stripe of the input data by each layer of the neural network, wherein the rows of data are processed sequentially horizontally according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network; and after processing the rows of data in the first stripe, the neural network Each layer of the network processes subsequent rows of data in a first stripe on a row-by-row basis, with each subsequent row of data being processed sequentially horizontally according to a layer-by-layer sequence; the neural network generates a first output stripe based on the processing of the data rows and subsequent rows of data; and after processing the first stripe, each layer of the neural network processes rows of data in a second stripe of input data on a row-by-row basis, with each row of data in the second stripe being processed horizontally according to a layer-by-layer sequence; and the neural network generates a second output stripe based on the processing of the second stripe.

[0007] In another exemplary example, a device for processing data in an inter-pixel neural network includes means for acquiring input data, wherein each stripe of the input data logically segments the input data, and each stripe contains at least one row of data; and means for processing rows of data in a first stripe of the input data by each layer of the neural network, wherein the rows of data are processed sequentially horizontally according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network; and after processing rows of data in the first stripe, by each layer of the neural network This may include means for processing subsequent rows of data in a first stripe on a row-by-row basis, wherein each subsequent row of data is processed sequentially in a horizontal direction according to a layer-by-layer sequence; a step of generating a first output stripe by a neural network based on the processing of rows of data and subsequent rows of data; means for processing rows of data in a second stripe of input data on a row-by-row basis by each layer of the neural network after processing the first stripe, wherein each row of data in the second stripe is processed horizontally according to a layer-by-layer sequence; and means for generating a second output stripe based on the processing of the second stripe.

[0008] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above may store in a first memory the output generated by a previous layer of a neural network for one or more blocks in a row of data within a first stripe, and in a second memory associated with a particular layer of the neural network one or more lines of data from one or more blocks in a row of data. In some examples, one or more lines of data may include portions of the data input of a particular layer of the neural network for a previous stripe-block-row.

[0009] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above generate inputs to a particular layer of a neural network based on combinations of one or more lines of data, including the output generated by the previous layer and the portion of the data input of the particular layer to the previous stripe-block-row, and the particular layer of the neural network may generate additional outputs based on the output to the previous layer.

[0010] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above may determine a portion of the input to a particular layer and store that portion of the input in a third memory associated with the particular layer of the neural network.

[0011] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above may generate additional inputs to subsequent layers of a neural network based on a combination of additional outputs of a particular layer and portions of the inputs to the subsequent layer from the preceding stripe-block-row of the subsequent layer, and the subsequent layers of the neural network may generate a second additional output based on the additional inputs to the subsequent layers.

[0012] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above may store a second additional output in a fourth memory. In some examples, the second and third memories may contain line stores within line store memory, and the first and fourth memories may contain buffers within scratch memory.

[0013] In some examples, processing rows of data within a first stripe horizontally sequentially may involve each layer of the neural network processing multiple blocks of data within a row sequentially. In some cases, each layer of the neural network processes each preceding block of data along the depth direction before processing subsequent blocks along the horizontal direction.

[0014] In some examples, processing subsequent rows may involve each layer of the neural network sequentially processing several blocks of data within each subsequent row. In some examples, each layer of the neural network processes the preceding blocks of data in the subsequent row along the depth direction before processing the subsequent blocks of data in the subsequent row.

[0015] In some examples, the neural network may include an inter-pixel neural network, and the input data may include pixels associated with an image.

[0016] In some examples, obtaining input data may involve logically segmenting the input data into stripes, where each stripe contains a portion of the input data.

[0017] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above may store the output generated by the previous layer of the neural network for one or more blocks in each subsequent row of data in a first memory associated with a particular layer of the neural network, and one or more lines of data from one or more blocks in each subsequent row of data in a second memory. In some examples, one or more lines of data may consist of one or more portions of one or more data inputs to a particular layer of the neural network.

[0018] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above generate inputs to a particular layer of a neural network based on combinations of one or more lines of data, including the output generated by the previous layer and one or more portions of one or more data inputs of a particular layer to a previous stripe-block-row; and the particular layer of the neural network may generate additional outputs based on the output from the previous layer and a subset of the inputs from the particular layer to the previous stripe-block-row.

[0019] In some embodiments, the methods, non-temporary computer-readable media, and apparatus described above may determine a portion of the input to a particular layer, store the input portion in a third memory associated with subsequent layers of the neural network, and store additional outputs in a fourth memory. In some cases, the second and fourth memories may include memory buffers.

[0020] In some examples, processing rows of data may involve each subsequent layer of the neural network processing the output of the preceding layer of the neural network, such that the output corresponds to a row of data.

[0021] In some examples, processing subsequent rows of data may involve processing the output of the preceding layer of the neural network by each subsequent layer of the neural network, such that the output corresponds to the subsequent rows of data in the first stripe.

[0022] In some embodiments, the device may be a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or so-called "smartphone," or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a 3D scanner, a multi-camera system, or other device, or a portion thereof. In some embodiments, the device includes one or more cameras for capturing one or more images. In some embodiments, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the device described above may include one or more sensors.

[0023] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0024] The above will become more apparent when considered in conjunction with other features and embodiments, with reference to the following specification, claims, and attached drawings.

[0025] Exemplary embodiments of the present application will be described in detail below with reference to the following figures.

Brief Description of the Drawings

[0026] [Figure 1] FIG. 1 is a simplified block diagram showing an exemplary image processing system according to some examples of the present disclosure. [Figure 2] FIG. 2 is a diagram showing an exemplary input to a pixel-wise neural network implementing a stripe and raster-based processing method according to some examples of the present disclosure. [Figure 3] FIG. is a diagram showing an exemplary data processing method for a neural network according to some examples of the present disclosure. [Figure 4] FIG. is a diagram showing an exemplary memory architecture for storing stripe data for generating layer inputs according to some examples of the present disclosure. [Figure 5A] FIG. is a diagram showing an exemplary data processing architecture for generating neural network inputs and outputs according to some examples of the present disclosure. [Figure 5B] FIG. is a diagram showing an exemplary data processing architecture for generating neural network inputs and outputs according to some examples of the present disclosure. [Figure 6] FIG. is a diagram showing an example of a layer input padded with a vertical line offset and used to generate a layer output according to some examples of the present disclosure. [Figure 7] FIG. is a diagram showing an exemplary funnel processing method for reducing redundant calculations when processing stripes according to various aspects of the present disclosure. [Figure 8] FIG. is a flowchart showing an exemplary process for processing data in a neural network according to some aspects of the present disclosure. [Figure 9] FIG. is a diagram showing an exemplary system architecture for implementing some of the aspects described herein.

MODE FOR CARRYING OUT THE INVENTION

[0027] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the sake of explanation, specific details are set forth in order to provide a complete understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0028] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments provides a description that enables the implementation of exemplary embodiments for those skilled in the art. It should be understood that various modifications may be made to the function and configuration of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0029] As mentioned above, neural networks can effectively perform a variety of complex tasks with accurate results. Often, neural networks can process large amounts of data, such as video, images, and / or audio data, to perform several tasks and produce specific outputs. The amount of data processed by a neural network and the processing demands of the neural network can place a considerable burden on device resources, such as memory, power resources, and computing resources. Some neural network architectures may be more resource-intensive than others and / or may require or utilize additional resources for their operation. This further increases the resource and / or processing costs of the neural network, reduces its efficiency, and / or, in some cases, increases its operational burden, resource burden, and requirements.

[0030] For example, inter-pixel neural networks (and other networks that handle other types of data similarly) generally use large memory buffers to temporarily store the data being processed by the inter-pixel network. However, the amount of data stored in the memory buffer and / or the memory buffer requirements can increase considerably, and even unsustainably, as the size of the input data increases. This can lead to increased hardware costs and hardware requirements. Resources in the device can limit the type, amount, and / or size of data that an inter-pixel neural network can handle. Often, even larger, more expensive memory buffers may not be able to support some inter-pixel neural network implementations and / or enable the inter-pixel neural network to handle certain types of data.

[0031] In some cases, the memory buffers used by inter-pixel neural networks can become oversubscribed. This can lead to increased demand for other resources, such as power and computational resources. For example, oversubscription of memory buffers used by an inter-pixel neural network can cause spikes in the bandwidth, processor clock cycles, and power used by the network. When memory buffers are oversubscribed and / or the inter-pixel neural network cannot store additional data in the memory buffers, performance metrics in the device can increase significantly. In some cases, such problems can be reduced or mitigated by increasing the amount and / or capacity of the device's hardware resources. For example, the memory resources and capacity in the device can be increased to handle the larger amounts of data used by the inter-pixel neural network. However, the amount of memory resources required to process certain types of data and / or perform certain tasks can be expensive, and even prohibitively costly.

[0032] As another example, a custom processor may be added to a device to handle some or all of the operations of an inter-pixel neural network. While a custom processor can increase the processing power of the device and support the additional processing demands of an inter-pixel neural network, an inter-pixel neural network may still require a large amount of memory to process the data. However, the memory resources required to support such a large amount of memory can be prohibitively expensive and / or in some cases, may hinder the implementation of a custom processor to address the memory requirements of an inter-pixel neural network. The memory and resource issues associated with inter-pixel neural networks can worsen in lower-level device implementations.

[0033] For example, in lower-end devices, the amount of memory resources used by inter-pixel neural networks is more limited. Often, the size of the memory buffers used by inter-pixel neural networks may need to be reduced. Generally speaking, the memory system or subsystem for transferring data to and / or from memory buffers may also need to be reduced. However, reducing memory buffers can cause spikes in bandwidth requirements, as previously discussed. This can be further exacerbated by reducing the memory system or subsystem, which can also reduce the amount of bandwidth available in the device. Lower-end devices are therefore disproportionately affected by reductions in memory and bandwidth.

[0034] This specification describes systems, apparatus, processes (also called methods), and computer-readable media (collectively referred to herein as “Systems and Techniques”) for efficiently processing data in neural networks and reducing resource burdens, including, in some embodiments, inter-pixel neural networks and any other neural networks that process large two-dimensional (2D) inputs (and / or three-dimensional (3D) inputs) into larger or greater two-dimensional (and / or three-dimensional) outputs (and / or 3D outputs). The Systems and Techniques described herein can increase neural network efficiency while reducing the resource and processing burden and cost of the neural network. In some examples, the Systems and Techniques may provide architectures that implement stripe and raster-based data processing schemes that enable the neural network to process and / or stream higher amounts of data between neural network arrays and reduce the amount of data stored in memory during the operation of the neural network. The Systems and Techniques described herein can reduce the amount of overlapping and / or redundant data processed by the neural network.

[0035] Examples of systems and techniques for processing data, as described in this specification, are shown in Figures 1 to 9 and described below. Figure 1 shows an exemplary computing system 100. The computing system 100 may implement a neural network for processing data. The neural network may implement architectures and data processing schemes to increase processing efficiency and / or reduce resource requirements, as described herein. The computing system 100 and / or the neural network may perform a variety of tasks and operations, such as extended reality, image / video processing and / or post-processing, data processing and / or post-processing, computer graphics, automation, machine vision, object modeling and object registration, multimedia rendering and / or configuration, and / or any other data processing tasks, effects, and / or tasks and operations for computation.

[0036] In some exemplary examples, computing system 100 implements one or more inter-pixel neural networks, as further described herein, and performs stripe and raster-based processing schemes and architectures. In some examples, the inter-pixel neural network may receive pixel inputs and produce pixel outputs. While various examples of this disclosure are described in relation to inter-pixel neural networks, the systems and techniques described herein may be implemented in and for other types and / or architectures of neural networks. For example, the systems and techniques described herein may be implemented for any neural network that processes a large 2D data input into a large or greater 2D data output, or any large 3D data input into a large or greater 3D data output. The inter-pixel neural network examples herein are non-limiting examples provided for illustrative and illustrative purposes only.

[0037] In the example shown in Figure 1, the computing system 100 includes one or more sensors 102, storage 104, memory 106, computing components 110, a processing engine 130, and a rendering engine 132. The computing system 100 may be a single computing device or part of multiple computing devices. In some examples, the computing system 100 may be part of an electronic device, such as a camera system (e.g., a digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., a smartphone, cellular phone, conferencing system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smartwatch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a game console, a video streaming device, a drone, an in-vehicle computer, a system-on-a-chip (SoC), an Internet of Things (IoT) device, or any other suitable electronic device.

[0038] In some implementations, one or more sensors 102, storage 104, memory 106, computing components 110, processing engine 130, and rendering engine 132 may be part of the same computing device. For example, in some cases, one or more sensors 102, storage 104, memory 106, computing components 110, processing engine 130, and rendering engine 132 may be integrated within a smartphone, laptop, tablet computer, smart wearable device, game system, server, SoC, and / or any other computing device. In other implementations, any one of the one or more sensors 102, storage 104, memory 106, computing components 110, processing engine 130, and / or rendering engine 132 may be part of two or more separate computing devices.

[0039] One or more sensors 102 may include one or more sensors, radar sensors, light detection and ranging (LIDAR) sensors, infrared (IR) sensors, gyroscopes, accelerometers, inertial measurement units (IMUs), audio sensors, and / or any other sensors. In some examples, one or more sensors 102 may include one or more image sensors. One or more image sensors may include image sensors and / or image capture devices capable of capturing images. For example, one or more image sensors may include one or more sensors capable of capturing red-green-blue (RGB) images. In some examples, an image (or frame) may include a red-green-blue (RGB) image or frame having red, green, and blue components per pixel, a luma, red difference, blue difference (YCbCr) image or frame having a luma component and two chroma (color) components (red difference and blue difference) per pixel, or any other preferred type of color or monochrome picture.

[0040] In some cases, one or more image sensors may be any type of image sensor (or image capture device) and / or video sensor (or video capture device). For example, one or more image sensors may include a digital camera sensor, a video camera sensor, a smartphone camera sensor, or an image / video capture device included as part of an electronic device (e.g., a television, computer, camera, etc.). In some cases, one or more image sensors may be part of a camera or computing device, such as a digital camera, video camera, IP camera, smartphone, smart television, game system, scanner, multi-camera system, or other computing device. For example, one or more image sensors may be part of a dual-camera system or device, such as a smartphone or camera. One or more image sensors may capture image data and / or video content (e.g., raw image and / or video data). The image data and / or video content may be processed by a computing component 110, a processing engine 130, and / or a rendering engine 132, as described herein.

[0041] Storage 104 may be any storage device for storing data, such as images, audio, files, or video data. Furthermore, storage 104 may also store data from any of the components of the computing system 100. For example, storage 104 may store data or measurements from one or more sensors 102, computing components 110 (e.g., parameters, inputs, outputs, generated images, calculation results, models, etc.), and / or data from either the processing engine 130 and / or the rendering engine 132 (e.g., output images, processing results, models, etc.). In some examples, storage 104 may include a buffer for storing data (e.g., image data) for processing by the computing components 110. Memory 106 may include any memory device for storing data, such as random access memory (RAM), read-only memory (ROM), a cache, or a memory buffer.

[0042] In some implementations, the computing component 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, an image signal processor (ISP) 118, and / or a neural processing unit (NPU) 120. In some examples, the computing component 110 may include one or more memories, such as a cache and / or memory buffers. The computing component 110 can perform a variety of operations, including data processing, computer vision, graphics rendering, extended reality (e.g., virtual reality, augmented reality, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, object recognition, feature recognition, face recognition, facial expression recognition, gaze recognition, age recognition, gender recognition, race recognition, tracking or pattern recognition, scene change recognition, etc.), machine learning, filtering, automation, computer graphics, and any of the various operations described herein. In some examples, the computing component 110 may implement a processing engine 130 and a rendering engine 132. In other examples, the computing component 110 may also implement one or more other processing engines.

[0043] The operations for the processing engine 130 and the rendering engine 132 may be implemented by one or more of the computing components 110. In one exemplary example, the processing engine 130 (and its associated operations) may be implemented by the CPU 112, DSP 116, ISP 118, and / or NPU 120, and the rendering engine 132 (and its associated operations) may be implemented by the GPU 114. In some cases, the computing components 110 may include other electronic circuits or hardware, computer software, firmware, or any combination thereof for performing any of the various operations described herein.

[0044] The computing component 110 can implement a processing engine 130 to perform various processing operations, such as image / video processing, XR rendering, computer vision, data processing, computer modeling, games, and computer graphics. The processing engine 130 can implement neural networks and processing architectures and / or schemes, as further described herein. For example, the processing engine 130 can implement an inter-pixel neural network and stripe and raster-based processing schemes, as described herein. In some examples, the processing engine 130 can be implemented by an NPU 120 to perform various tasks using stripe and raster-based processing patterns, as described herein.

[0045] While computing system 100 is shown as including several components, those skilled in the art will understand that computing system 100 may include more or fewer components than those shown in Figure 1. For example, computing system 100 may also optionally include one or more additional memory devices (e.g., RAM, ROM, cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 1. Exemplary examples of computing devices and hardware components that can be implemented by computing system 100 are further described below with respect to Figure 9.

[0046] As mentioned above, some neural network architectures, such as inter-pixel neural network architectures, use or require large memory buffers to process data. Memory buffer size can increase unsustainably as the size of the input data increases. Even large memory buffers may not be sufficient for some data processing scenarios. When memory buffers are oversubscribed and / or full, other resource requirements and / or utilization can increase substantially. For example, when memory buffers are oversubscribed, bandwidth, power, clock cycles, etc., also increase substantially. Adding more memory buffers and / or memory buffers with higher capacity can be expensive and may not adequately prevent and / or mitigate memory buffer oversubscription and / or increased other resource requirements and / or utilization. The cost of large memory buffers may also prevent the addition of other hardware resources, such as custom processors, to address such resource needs and / or limitations.

[0047] The systems and techniques described herein provide architectures and data processing schemes to address these and other challenges / limitations. These architectures and data processing schemes can be used to process any type of data, including image data, video data, audio data, and files. For illustrative purposes, the architectures and data processing schemes will be described in relation to processing image data.

[0048] In some examples, the systems and techniques described herein may provide stripe and raster-based processing schemes. Stripe and raster-based processing schemes may divide the input to a neural network into vertical stripes, which the neural network processes according to specific patterns as described below. Each layer of the neural network may process the data in stripes along the depth and horizontal directions before moving vertically to process additional data. Additional data may similarly be processed along the depth and horizontal directions before moving vertically to process new data.

[0049] Figure 2 shows an exemplary input 200 for an interpixel neural network to implement stripe and raster-based processing schemes. The input 200 may include, for example, images, audio files, 2D data, 3D data, and / or any other type of data. In this example, the input 200 represents an image being processed by an interpixel neural network, such as a conventional interpixel neural network. The input 200 includes stripes 210A and 210B, as will be further described herein.

[0050] In Figure 2, the input 200 is shown in Figure 230A on the left and Figure 230B on the right, showing stripes 210A and 210B at different locations within the input 200. Figure 230A on the left shows stripe 210A in the left portion of the input 200 during a first processing time, and Figure 230B on the right shows stripe 210B in the right portion of the input 200 during a second processing time. The input 200 is shown divided in Figures 230A and 230B to show different stripes 210A and 210B within the input 200 at different times. While the example in Figure 2 shows two stripes, in other examples, the input 200 may contain more or fewer stripes than shown in Figure 2. In some examples, the number of stripes in the input 200 may vary or depend on the neural network processing requirements and / or architecture.

[0051] Stripes 210A and 210B may each contain several stripe blocks extending vertically from the top to the bottom of input 200. For example, Figure 2 shows stripe block 212A at the top of stripe 210A and stripe block 212B at the bottom of stripe 210A, as well as stripe block 220A at the top of stripe 210B and stripe block 220B at the bottom of stripe 210B. Each of these stripe blocks may represent a row (or layer) of stripes 210A and 210B. Although not shown in Figure 2 for simplicity, stripe 210A may contain an additional stripe block extending from stripe block 212A to stripe block 212B, just as stripe 210A contains a row (or layer) of stripe blocks from top to bottom. Similarly, although not shown in Figure 2 for simplicity, stripe 210B may include additional stripe blocks extending from stripe block 220A to stripe block 220B, so that stripe 210B includes rows (or layers) of stripe blocks from top to bottom.

[0052] The input 200 and the stripes 210A and 210B may include a horizontal dimension 202 (e.g., along the x-axis), a vertical dimension 204 (e.g., along the y-axis), and a depth dimension 206 (e.g., along the z-axis). Each layer of the neural network may process the data in the stripe along the depth dimension 206 and horizontal dimension 202 before processing the data along the vertical dimension 204.

[0053] Stripe 210A may include overlapping region 216, and stripe 210B may include overlapping region 222. Overlapping regions 216 and 222 may contain an amount of overlap in the input processed by the neural network. As shown, the overlap in overlapping regions 216 and 222 is in the horizontal dimension 202. Any overlap in the vertical dimension 204 may be stored in memory for use as needed by the neural network. In some cases, the overlap may result in a certain amount of extra and / or redundant computation. However, the configurations of stripes 210A and 210B and the data processing schemes described herein can avoid overlap in other directions. Therefore, the amount of overlap in the input 200 may be less than the amount of overlap in other data processing schemes and architectures, resulting in less extra and / or redundant computation.

[0054] For example, the configurations and data processing methods of stripes 210A and 210B described herein can avoid duplication on at least one side along the horizontal dimension 202, at least one side along the vertical dimension 204, and one or both sides along the depth dimension 206 of the stripe blocks (e.g., stripe block 212A, stripe block 212B, stripe block 220A, stripe block 220B). This can significantly reduce the memory and processing requirements and / or burden of the neural network. Furthermore, the memory and processing requirements and / or burden of the neural network can be further reduced by including only a small amount of duplication along one side of the horizontal dimension 202 (relative to the stripe block) while processing and storing any amount of duplication that may exist along one side of the vertical dimension 204 (relative to the stripe block).

[0055] Figure 3 shows an exemplary data processing scheme 300 for a neural network 310. The data processing scheme 300 may include a stripe-raster data processing sequence or pattern. The neural network 310 may implement the data processing scheme 300 to process data within stripes 210A and 210B. The neural network 310 may include multiple layers (e.g., layers 312-318) for processing input data and generating outputs.

[0056] The neural network 310 may include any neural network that takes a large n-dimensional input and produces a large or larger n-dimensional output. For example, in some cases, the neural network 310 may include an inter-pixel neural network. The inter-pixel neural network may process input pixels from an image and produce output pixels for the image. In some examples, the image may be a large and / or high-resolution image. The image may be, for example, a 2D image or a 3D image. In other cases, the neural network 310 may include a neural network from 2D input to 2D output. For example, the neural network 310 may produce a 2D output from a 2D input. In some examples, the neural network 310 may process and / or transform each part of the 2D data and produce a 2D data output for each part of the 2D data. The 2D input and 2D output may include, for example, an image, video data, audio data, or any other 2D data.

[0057] In other cases, the neural network 310 may include a neural network that takes a 3D input and outputs a 3D output. For example, the neural network 310 may generate a 3D output from a 3D input. In some examples, the neural network 310 may process and / or transform each part of the 3D data to generate a 3D data output for each part of the 3D data. The 3D input and 3D output may include, for example, images, video data, audio data, or any other 3D data.

[0058] In Figure 3, stripes 210A and 210B include block rows 306A-N for processing data within stripe blocks 302 according to the data processing scheme 300. Stripe block 302 may include tiles 304 (or subblocks) of data that are first processed along the depth dimension 206 of stripe block 302 before shifting stripe block 302 along the horizontal dimension 202 in order to process data within tiles 304 along the depth dimension 206 of stripe block 302, which is shifted along the horizontal dimension 202. Tiles 304 may continue to be processed along the depth dimension 206 of stripe block 302, and stripe block 302 may continue to be processed along the horizontal dimension 202 (for example, across corresponding block rows). Once all layers 312-318 have processed data across block rows, the processing scheme 300 may move along the vertical dimension 204 to similarly process data in the next row of the stripe.

[0059] Each layer of the neural network 310 (for example, each of layers 312-318) may process the data in the stripe block 302 (and rows) according to the pattern described above, before the next layer in the neural network 310 processes the data in the stripe block 302 (and rows) according to the same processing order / pattern. For example, each layer in the neural network 310 may process the data in the stripe block 302 in row 306A along the depth dimension 206 of the stripe block 302, followed by the horizontal dimension 202 of row 306A, before the next layer in the neural network processes the data in the stripe block 302 in a similar pattern.

[0060] For illustrative purposes, layer 312 of the neural network 310 (e.g., layer 0) may first process the tiles 304 of stripe block 302 in row 306A of stripe 210A, along the depth dimension 206 of the stripe block 302. Once layer 312 has processed the data in stripe block 302 in row 306A, layer 312 may move along the horizontal dimension 202 and process the data in adjacent stripe blocks in row 306A. Before layer 312 moves again along the horizontal dimension 202 of row 306A to the next adjacent stripe block, layer 312 may process the tiles in adjacent stripe blocks along the depth dimension 206. Layer 312 can continue processing data in this pattern until it has finished processing the data in row 306A. Once layer 312 has finished processing the data in row 306A, the next layer in the neural network 310, layer 314 (e.g., layer 1), may process the data in row 306A according to the same pattern. For example, layer 314 can process data within a striped block sequentially along the depth dimension, and blocks within row 306A can process data sequentially along the horizontal dimension 202. Once layer 314 has finished processing the data in row 306A, the next layer of the neural network 310, layer 316 (for example, layer 2), can process the data in row 306A according to the same pattern. Once layer 314 has finished processing the data in row 306A, the next layer of the neural network 310, layer 318 (for example, layer N), can process the data in row 306A according to the same pattern.

[0061] In this way, each layer in the neural network 310 can sequentially process the data in row 306A according to the pattern described above before moving along the vertical dimension 204 of stripe 210A to process the data in the next row, row 306B, similarly (for example, according to the same pattern). Once all layers in the neural network 310 have finished processing the data in row 306B, the neural network 310 can move along the vertical dimension 204 of stripe 210A, and each layer in the neural network 310 can similarly process the data in the next row of stripe 210A. The neural network 310 can continue moving along the vertical dimension 204 until the data in the last row of stripe 210A (for example, row 306N) has been processed by each layer.

[0062] Once stripe 210A is processed, the neural network 310 may process the next stripe, stripe 210B. Until the neural network 310 has finished processing all the data in stripe 210B, it may process stripe 210B in the order / pattern described above with respect to stripe 210A. As previously described, the input data may contain more or fewer stripes than those shown in Figures 2 and 3. Therefore, if the input data contains additional stripes, the neural network 310 may proceed to the next stripe and process the data in the next stripe, as previously described. The neural network 310 may continue to process any additional stripes until all stripes have been processed.

[0063] Each stripe (e.g., stripes 210A and 210B) may contain any number of rows. Furthermore, the size of stripe blocks (e.g., stripe block 302) and tiles (e.g., tile 304) is not limited to any particular size and may vary in different implementations. In some examples, the size of stripe blocks and / or tiles may be based on preferences, neural network architecture, data path architecture, neural network requirements, input type and / or attributes, stripe size and / or number, and / or any other factors. For example, in some cases, the size of stripe blocks and / or tiles may be based on a multiplication-cumulative (MAC) data path architecture.

[0064] In some examples, the data processing scheme 300 may include an internal processing loop and an external processing loop. The internal processing loop may process the data in the stripe block according to the pattern described earlier. In some examples, the internal processing loop may process the horizontal blocks (or subtiles) in the stripe block sequentially or in parallel, the vertical blocks (or subtiles) in the stripe block sequentially or in parallel, the tile output depth sequentially or in parallel, and the input depth sequentially or in parallel. In one exemplary example, the horizontal blocks, vertical blocks, and tile output depth may be "unrolled" (e.g., processed and / or output in parallel), and the input depth may be "walked" (e.g., processed and / or output sequentially).

[0065] The outer loop can process the input data in a specific order, as previously described. For example, the outer loop can process the data in the depth direction, followed by the horizontal direction, followed by the neural network layer order, followed by the vertical direction, and then by the stripe order. For example, the first layer of the neural network may first process the data within a stripe block along the depth dimension of the stripe block. The first layer of the neural network may then move horizontally to the next stripe block and process the data within that stripe block in the same way. The first layer of the neural network can continue to move horizontally and process the data within each stripe block horizontally until all blocks in the row have been processed. The second layer of the neural network may then perform the same pattern to process the stripe blocks in the row. Each subsequent layer in the neural network may perform the same pattern in the same way until all layers have processed the stripe blocks in the row. Once all layers have processed the stripe blocks in the row, the first layer of the neural network may move vertically along the stripe to process the stripe blocks in the next row in the same way. Each layer of the neural network can sequentially process stripe blocks in the next row, and the neural network can continue moving horizontally until all layers have processed all stripe blocks in all rows. The neural network can then proceed to any subsequent stripe and process subsequent stripes sequentially according to the outer loop order described above.

[0066] Figure 4 shows an exemplary memory architecture 400 for storing stripe data to generate layer inputs. In this example, the memory architecture 400 includes line store memory 402 and scratch memory 420. In some examples, the line store memory 402 and scratch memory 420 may include one or more random access memories.

[0067] The scratch memory 420 can store outputs from layers of the neural network 310 (e.g., layers 312-318). For example, buffer 422 in the scratch memory 420 can store the output from layer 0 (e.g., layer 312), and buffer 424 in the scratch memory 420 can store the output from layer 1 (e.g., layer 314). When additional layers of the neural network generate outputs, those outputs can similarly be stored in the scratch memory 420. Outputs from layers in the neural network can be used as inputs for the next layer in the neural network. In some examples, the scratch memory 420 may contain buffers for each output of each layer of the neural network 310. In some cases, buffers in the scratch memory (e.g., buffers 422 and 424) may contain ping-pong buffers.

[0068] The line store memory 402 may store portions of data from layers within the neural network 310 (e.g., data lines, data blocks, data windows, subsets of data rows from striped blocks, etc.). The data stored in the line store memory 402 may include overlapping data from two or more layers of the neural network 310 (e.g., overlapping regions 216 and 222). In some examples, the line store memory 402 may include line stores 404-410. Each line store may correspond to a respective layer of the neural network 310, and the data in a line store may be used as part of the input to the layer associated with that line store. In this example, line store 404 may correspond to layer 0 (e.g., layer 312) of the neural network 310, line store 406 may correspond to layer 1 (e.g., layer 314) of the neural network 310, line store 408 may correspond to layer 2 (e.g., layer 316) of the neural network 310, and line store 410 may correspond to layer N (e.g., layer 318) of the neural network 310. The data stored in the line stores associated with the layers of the neural network may include a subset of data from previous layers of the neural network 310 (e.g., one or more lines, one or more data blocks, input windows, a subset of data rows from stripe blocks).

[0069] For example, line store 406 corresponding to layer 1 (e.g., layer 314) of neural network 310 may contain a subset of output data from layer 0 (e.g., layer 312) of neural network 310. In some cases, the size of the stored data may be based on the size of the convolutional filters implemented by neural network 310. For example, convolutional filters may introduce some shifts within the output data from each layer. Since a layer uses the output from the previous layer as part of its input, the shifts in the output data may reduce the size of the input for that layer. The portion of data from the previous layer stored in the line store (e.g., duplicates) may then be used as part of the input to the next layer in order to increase the size of the input to the next layer. For illustrative purposes, convolutional filters implemented by the layers of neural network 310 may introduce downscaling of the output data. Therefore, the downscaled output from a layer may need to be increased to construct the input to the next layer.

[0070] As described above, line store 406 corresponding to layer 1 (e.g., layer 314) of neural network 310 may contain a subset of output data from layer 0 (e.g., layer 312) of neural network 310. Similarly, line store 408 corresponding to layer 2 (e.g., layer 316) of neural network 310 may contain a subset of output data from layer 1 (e.g., layer 314) of neural network 310, and line store 410 corresponding to layer N (e.g., layer 318) of neural network 310 may contain a subset of output data from layer 2 (e.g., layer 316) of neural network 310. In some examples, since layer 0 is the first layer and does not have data from previous layers, the data stored in line store 404 corresponding to layer 0 may include padding. Padding may include, for example, edge duplication (e.g., repeating the first one or more input lines), value duplication, and / or any other padding techniques and / or data.

[0071] In some examples, duplicate data from one layer of the neural network 310 may include vertically duplicated portions from the data processing scheme 300. For example, as previously described, the data processing scheme 300 may include horizontally duplicated portions (e.g., duplicate regions 216 and 222). In some cases, horizontal duplicates may be processed by each layer of the neural network 310 as each layer processes data within a row of the stripe (e.g., rows 306A, 306B, 306N). Additional downward vertical duplicates may be stored in the line store memory 402 for use when processing the next row in the stripe. In some cases, this vertical duplicate may include the duplicated data described above with respect to line stores 404-410. For example, the duplicated data may include vertically duplicated portions of data from the previous row of the stripe processed by the neural network 310.

[0072] The memory architecture 400 can prepare and process layer data using output stored in scratch memory 420 and duplicate data in line store memory 402. For example, in Figure 4, the output 456 of layer 0 of the neural network 310 is stored in buffer 422, and the duplicate data 454 from layer 0 of the neural network 310 is stored in line store 406. To construct the input 452 for layer 1, the output 456 of layer 0 from buffer 422 may be combined with the duplicate data 454 from line store 406. The combination of the output 456 of layer 0 and the duplicate data 454 may result in the input 452 for layer 1.

[0073] As shown, the memory architecture 400 can combine data from two memories (for example, line store memory 402 and scratch memory 420) on a layer-by-layer basis to form input data for any given layer. In some cases, the layers are processed sequentially without the need to fetch data from system memory.

[0074] In some examples, line store memory 402 may store m lines of input data for each neural network layer. For example, line store 404 may store m lines of input data for layer 0, line store 406 may store m lines of input data for layer 1, line store 408 may store m lines of input data for layer 2, and line store 410 may store m lines of input data for layer N. In some cases, the number of m lines may depend on the kernel size of the associated neural network layer.

[0075] Each buffer in scratch memory 420 may store the output of a particular neural network layer. In some cases, the output of a particular neural network layer stored in a buffer may contain n lines of data. For example, buffer 422 may store the output of layer 0 containing n lines of data, and buffer 424 may store the output of layer 1 containing n lines of data.

[0076] In some cases, n lines of data may be passed from layer to layer. In some examples, n lines of data may differ in different examples and / or depend on the architecture used. In some cases, increasing the number of lines n may enable greater MAC parallelism. For example, if the MAC engine is 1024 MAC / cycle, the MAC engine may process several lines (e.g., n lines), several pixels (or other types of data), and several input and output depths simultaneously. In some cases, these parameters may be combined to form the internal loop tile size.

[0077] Figure 5A shows an exemplary data processing architecture 500 for generating neural network layer inputs and outputs. The data processing architecture 500 includes line store memory 402 and scratch memory 420. As previously described, scratch memory 420 may store outputs from layers of the neural network 310 (e.g., layers 312-318). For example, buffer 422 in scratch memory 420 may store the output from layer 0 (e.g., layer 312), and buffer 424 in scratch memory 420 may store the output from layer 1 (e.g., layer 314). As additional layers of the neural network generate outputs, those outputs may similarly be stored in scratch memory 420. Outputs from layers in the neural network can be used as inputs for the next layer in the neural network. In some examples, scratch memory 420 may contain the same number of buffers as there are buffers for each output of each layer of the neural network 310. In another example, scratch memory 420 may contain a buffer equivalent to only two and a half block-row-layer outputs.

[0078] The line store memory 402 may store portions of data from layers within the neural network 310 (e.g., data lines, data blocks, data windows, subsets of data rows from striped blocks, etc.). The data stored in the line store memory 402 may include overlapping data from two or more layers of the neural network 310 (e.g., overlapping regions 216 and 222). In some examples, the overlapping data from one layer of the neural network 310 may include portions of vertical overlap from the data processing scheme 300.

[0079] The data processing architecture 500 can prepare and process layer data using output stored in scratch memory 420 and duplicate data in line store memory 402. For example, in Figure 5A, the output 540 of layer 0 of the neural network 310 is stored in buffer 422, and duplicate data 542 from a previous run of layer 1 of the neural network 310 is stored in line store 406. To prepare and process data for layer 1, the vertical composer 530 may retrieve the output 540 of layer 0 from buffer 422 and the duplicate data 542 from line store 406 to generate data 544 for layer 1 of the neural network 310. To generate data 544, the vertical composer 530 may combine the output 540 and the duplicate data 542. In some examples, data 544 may include inputs constructed for layer 1.

[0080] Data 544 may be processed through the MAC data path 532 to generate Layer 1 output 548. In some examples, output 548 may contain blocks of data corresponding to stripe blocks within a row of stripes. The data blocks may contain data along the depth dimension of the stripe block, from its horizontal position within the row of stripes. In other examples, output 548 may contain Layer 1 data outputs corresponding to rows of stripes.

[0081] The output 548 of Layer 1 can then be stored in a buffer 424 on scratch memory 420 for use by Layer 2. In some examples, the MAC data path 532 may include a MAC array that can perform matrix multiplication operations to generate the output 548. In some cases, the MAC data path 532 may use a convolution kernel of Layer 1 to generate the output 548. For example, the convolution kernel may convolve the data to generate the output 548.

[0082] The line store update component 534 may retrieve the data 544 generated by the vertical composer 530 in order to generate duplicate data 546 for the next row of layer 1. The line store update component 534 may then store the duplicate data 546 in the line store 406 associated with layer 1. In some examples, the vertical composer 530 may determine and / or cut portions of the data 544 in order to generate the duplicate data 546. In some cases, the vertical composer 530 may determine and / or cut the bottom portion of the data 544 in order to generate the duplicate data 546. The data processing architecture 500 may then prepare and process the data for layer 2 using the duplicate data 546 in the line store 408 and the output 548 in the buffer 424 from when layer 2 was previously executed.

[0083] In some examples, the data processing architecture 500 may optionally include a horizontal composer 560 for adding horizontally duplicated data 550 to data 544 generated by the vertical composer 530. For example, referring to Figure 5B, the horizontal composer 560 may take data 544 from the vertical composer 530 and add horizontally duplicated data 550 to data 544 to generate a block input 552 to the MAC array 562. The block input 552 may be used as a MAC array input to the MAC array 562. The block input 552 may include data 544 and horizontally duplicated data 550 added to or appended to the horizontal edges of data 544.

[0084] In some examples, horizontally duplicated data 550 may pad data 544 with horizontal data (e.g., horizontally duplicated data 550) prior to data 544 being processed by MAC array 562. In some cases, by adding horizontally duplicated data 550 to data 544, the data processing architecture 500 may reduce the total bandwidth used by the data processing architecture 500 and data processing scheme 300. In some examples, horizontally duplicated data 550 may include a subset of data from a previous horizontal location processed within the same row of the stripe as data 544.

[0085] The MAC array 562 may use the block input 552 to generate output 548, which can be stored in buffer 424 as previously described. In some examples, the MAC array 562 may perform matrix multiplication to generate output 548. In some cases, the horizontal composer 560 may implement the IM2COL operation on data 544 and horizontal duplicate data 550 to generate block input 552.

[0086] As shown in Figure 5B, the data processing architecture 500 may include a write controller 570 for storing data in line store memory 402 and a read controller 572 for reading / retrieving data from line store memory 402. Similarly, the data processing architecture 500 may include a write controller 574 for storing data in scratch memory 420 and a read controller 576 for reading / retrieving data from scratch memory 420.

[0087] Figure 6 shows an example of a layer input 602, which is padded with a vertical line offset and used to generate the layer output 610. In this example, the layer input 602 corresponds to the first pass of a particular neural network layer. The layer input 602 includes an input line 606 and padding 604. In some examples, the input line 606 may contain data retrieved from scratch memory 420, such as the layer output stored in a buffer within scratch memory 420.

[0088] In some cases, padding 604 may be added before and / or above the first line 608 of the input line 606. Padding 604 may provide a vertical line offset. In some examples, the vertical line offset may be used to increase the size of the layer input to accommodate the shift and / or downscaling caused by the convolution filter as described earlier. In some cases, padding 604 may contain duplicate data retrieved from line store memory 402. Duplicate data may contain one or more vertical lines from previous stripe rows processed by the layer.

[0089] In some cases, if the layer associated with layer input 602 is layer 0 (for example, layer 312) and no duplicate data from previous processing is available, padding 604 may include repeating lines and / or values ​​used to pad layer input 602. For example, padding 604 may include one or more lines from the first line 608 and / or input line 606.

[0090] The layer input 602 may be used to generate the layer output 610. The layer output 610 may include an output line 612. In some examples, the output line 612 may be stored in a buffer in scratch memory 420 for use in constructing the input for the next neural network layer.

[0091] As shown in Figure 6, the layer input 602 is larger than the layer output 610. In some examples, convolution within a neural network layer can cause shifts and / or reductions in the data. These shifts and / or reductions can cause the layer output 610 to be smaller than the layer input 602. Therefore, when the layer output 610 is used to generate the input for the next layer, similar padding (e.g., padding 604) of the layer output 610 may be used to increase the size of the input for the next layer. In some cases, duplicate data (e.g., one or more vertical lines from the previous layer and / or row) may be used for padding the layer output 610.

[0092] Figure 7 shows an exemplary funnel-type processing scheme 700 for reducing redundant calculations when processing stripes. In this example, the central stripe 710 includes horizontal overlaps 712 on each side of the central stripe 710. The horizontal overlaps 712 are used to generate output stripes 708 corresponding to the central stripe 710. In this example, the output stripes 708 are generated using layers 0 to N. A neural network (e.g., neural network 310) can process the stripe data layer by layer through layers 0 to N.

[0093] Horizontal duplication 712 can increase the number of operations performed by the neural network (and / or layers of the neural network) and the burden on system resources, such as bandwidth, memory, power, and / or computation, which can increase if each additional layer is implemented by a neural network. In some examples, the more layers implemented by the neural network, the greater the burden on resources and the number of redundant operations performed by the layers of the neural network.

[0094] The funnel-type processing method 700 can reduce resource load and redundant computations by progressively reducing the active stripe width as the neural network progresses layer by layer. In some examples, the funnel-type processing method 700 can also reduce the number of MACs per layer.

[0095] The funnel-type processing method 700 may first generate an input stripe 702 for layer 0. As shown in Figure 7, the input range of the input stripe 702 for layer 0 is larger than the input ranges for layers 1, 2, and N. For example, when moving from layer 0 to layer 1, the funnel-type processing method 700 may reduce the input range of the intermediate stripe 704 for layer 1 compared to the input range of the input stripe 702 for layer 0. Similarly, the funnel-type processing method 700 may reduce the input range of the intermediate stripe 706 for layer 2 compared to the input range of the intermediate stripe 704 for layer 1. The funnel-type processing method 700 may further reduce the input range for the output stripe 708 at layer N compared to the input range of the intermediate stripe 706 for layer 2.

[0096] In this way, the funnel-type processing method 700 can novelly reduce the input range for each layer of the neural network, as well as the redundant calculations used to generate the output stripe 708. The reduction in computation and processing data can reduce the burden on resources such as memory, computation, power, and bandwidth.

[0097] Figure 8 is a flowchart illustrating exemplary processes 800 for processing data in a neural network, according to some examples of the present disclosure. In block 802, process 800 may include obtaining input data (e.g., input 200) which includes stripes (e.g., stripe 210A, stripe 210B) that logically segment the input data. Each stripe of the input data includes at least one row of data, such as stripe 210A shown in Figure 2. In some examples, process 800 may include logically segmenting the input data into stripes. In some cases, each stripe may include the respective portion of the input data.

[0098] In block 804, process 800 may include processing rows of data (e.g., row 306A) within a first stripe of input data (e.g., stripe 210A) by each layer (e.g., layer 312, layer 314, layer 316, layer 318) of the neural network (e.g., neural network 310). In some examples, rows of data may be processed sequentially according to a layer-by-layer sequence, where each preceding layer of the neural network processes a particular row of data (e.g., row 306A in stripe 210A, or a row of data in the first stripe) before each subsequent layer of the neural network, in the horizontal direction (e.g., horizontal dimension 202).

[0099] In some cases, processing rows of data sequentially horizontally may involve each layer of the neural network sequentially processing multiple blocks of data within a row (e.g., striped blocks 302). In some cases, each layer of the neural network may process each preceding block of data along the depth direction (e.g., depth dimension 206) before processing subsequent blocks along the horizontal direction (e.g., horizontal dimension 202).

[0100] In some cases, a neural network may include a convolutional network. In some examples, a neural network may include an inter-pixel neural network, where the input data may include pixels associated with an image. In other cases, a neural network may include one that processes 2D data input into 2D data output and / or 3D data input into 3D data output.

[0101] In some examples, processing rows of data according to a layer-by-layer sequence may involve each subsequent layer of the neural network processing the output of the preceding layer of the neural network. In some examples, the output may correspond to rows of data.

[0102] In block 806, process 800 may include processing subsequent rows of data in the first stripe (e.g., rows 306B, 306N) on a row-by-row basis, after each layer of the neural network has processed a row of data in the first stripe. In some examples, each subsequent row of data may be processed horizontally, sequentially according to a layer-by-layer sequence (e.g., each preceding layer of the neural network processes a subsequent row before each subsequent layer of the neural network).

[0103] In some cases, processing subsequent rows may involve each layer of the neural network sequentially processing several blocks of data within each subsequent row (e.g., stripe blocks 302). In some cases, each layer of the neural network may process preceding blocks of data in subsequent rows within the first stripe along the depth direction before processing subsequent blocks of data in subsequent rows.

[0104] In some cases, processing subsequent rows of data according to a layer-by-layer sequence may involve each subsequent layer of the neural network processing the output of the preceding layer of the neural network. In some examples, the output may correspond to subsequent rows of data.

[0105] In block 808, process 800 may include generating a first output stripe based on the processing of a row of data and subsequent rows of data using a neural network.

[0106] In block 810, process 800 may include processing rows of data in a second stripe (e.g., stripe 210B) of the input data after processing the first stripe by each layer of the neural network. In some examples, rows of data in the second stripe may be processed on a row-by-row basis. In some examples, each row of data in the second stripe may be processed horizontally, according to a layer-by-layer sequence.

[0107] In block 812, process 800 may include generating a second output stripe based on the processing of the second stripe. In some examples, the neural network may generate an output for the input data based on the first output stripe and / or the second output stripe.

[0108] In some cases, process 800 may include storing the output (e.g., output 548) generated by a previous layer of the neural network for one or more blocks within a row of data (e.g., stripe block 302) in a first memory (e.g., scratch memory 420), and storing one or more lines of data from one or more blocks within a row of data in a second memory (e.g., line store memory 402) associated with a particular layer of the neural network. In some examples, one or more lines of data may include portions of the data input of a particular layer of the neural network for a previous stripe-block-row (e.g., duplicate data 542, duplicate data 546).

[0109] In some examples, process 800 may include generating an input to a particular layer of the neural network (e.g., input 452, block input 552) based on a combination of one or more lines of data that include the output generated by the previous layer and the portion of the data input to this same layer for the previous stripe-row. Process 800 may also include the step of generating an additional output by the particular layer of the neural network based on the input to the previous layer.

[0110] In some examples, process 800 may include determining and / or cutting off portions of the input to a particular layer (e.g., duplicate data 542, duplicate data 546). Process 800 may also include storing portions of the input in a third memory associated with the particular layer of the neural network (e.g., line store memory 402).

[0111] In some cases, process 800 may include generating additional inputs to subsequent layers of the neural network based on a combination of additional outputs from a particular layer and portions of the inputs from the previous block-row of the subsequent layer. Process 800 may also include the subsequent layers of the neural network generating a second additional output based on the additional inputs to the subsequent layers.

[0112] In some cases, process 800 may include storing a second additional output in a fourth memory (e.g., scratch memory 420). In some examples, the second and third memories may contain line stores (e.g., line stores 404, 406, 408, and 410) within line store memory (e.g., line store memory 402), and the first and fourth memories may contain buffers (e.g., buffers 422 and 424) within scratch memory (e.g., scratch memory 420).

[0113] In some cases, process 800 may include storing the output generated by the previous layer of the neural network for one or more blocks in each subsequent row of data in a first memory (e.g., scratch memory 420) associated with a particular layer of the neural network. Process 800 may also include storing one or more lines of data from one or more blocks in each subsequent row of data in a second memory (e.g., line store memory 402). In some examples, one or more lines of data may include one or more portions of one or more data inputs to a particular layer of the neural network.

[0114] In some cases, process 800 may include generating inputs to a particular layer of the neural network based on a combination of one or more lines of data that include the output generated by the previous layer and one or more portions of one or more data inputs of the particular layer to the previous stripe-block-row. Process 800 may also include the particular layer of the neural network generating additional outputs based on the output from the previous layer and a subset of the inputs from the particular layer to the previous stripe-block-row.

[0115] In some cases, process 800 may include determining and / or cutting off portions of the input to a particular layer. Process 800 may include storing portions of the input in a third memory (e.g., line store memory 402) associated with subsequent layers of the neural network. Process 800 may further include storing additional outputs in a fourth memory (e.g., scratch memory 420). In some examples, the second and fourth memories may include memory buffers (e.g., buffers 422, buffers 424).

[0116] In some examples, the processes described herein (e.g., process 800, and / or other processes described herein) may be performed by a computing device or apparatus. In one example, process 800 may be performed by the computing system 100 shown in Figure 1. In another example, process 800 may be performed by a computing system having the computing device architecture 900 shown in Figure 9. For example, a computing device having the computing device architecture 900 shown in Figure 9 may implement the operation of Figure 8 and / or the components and / or operations described herein with respect to any of Figures 1 to 8.

[0117] Computing devices may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a laptop computer, a smart television, a camera, and / or any other computing device having the resources to perform the processes described herein, including process 800 and / or any other processes described herein. In some cases, a computing device or apparatus may include a variety of components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0118] Components of a computing device can be implemented within a circuit configuration. For example, a component may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), electronic circuits, or other electronic hardware, and / or can be implemented using them, and / or may include computer software, firmware, or any combination thereof, and / or can be implemented using them to perform the various operations described herein.

[0119] Process 800 is presented as a logical flow diagram, and its operation represents a set of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be interpreted as limiting, and any number of operations described may be combined in any order and / or in parallel to implement the process.

[0120] In addition, process 800, and / or other processes described herein, may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or in combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0121] Figure 9 shows an exemplary computing device architecture 900 of an exemplary computing device that can implement various techniques described herein. For example, the computing device architecture 900 can implement at least some parts of the computing system 100 shown in Figure 1. The components of the computing device architecture 900 are shown as communicating electrically with one another using connections 905 such as buses. The exemplary computing device architecture 900 includes a processing unit (CPU or processor) 910 and computing device connections 905 that connect various computing device components, including computing device memory 915 such as read-only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910.

[0122] The computing device architecture 900 may include a cache of high-speed memory directly connected to, near, or integrated as part of the processor 910. The computing device architecture 900 may copy data from memory 915 and / or storage device 930 to the cache 912 for high-speed access by the processor 910. In this way, the cache can provide performance improvements by eliminating delays for the processor 910 while waiting for data. These and other modules may control, or be configured to control, the processor 910 to perform various actions. Other computing device memories 915 may also be available. Memory 915 may include multiple different types of memory with different performance characteristics. The processor 910 may include any general-purpose processor, and hardware or software services (e.g., service 1 932, service 2 934, and service 3 936) stored in the storage device 930 and configured to control the processor 910, as well as dedicated processors such that software instructions are incorporated into the processor design. The processor 910 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multicore processors can be symmetrical or asymmetrical.

[0123] To enable user interaction with the computing device architecture 900, the input device 945 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, or speech. The output device 935 may be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, or speaker device. In some cases, a multimodal computing device may allow the user to provide multiple types of input to communicate with the computing device architecture 900. The communication interface 940 can generally control and manage user input and computing device output. Since there are no constraints that act on any particular hardware configuration, this basic feature may be easily superseded by improved hardware or firmware configurations as development progresses.

[0124] The storage device 930 is non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic cassette, flash memory card, solid-state memory device, digital versatile disk, cartridge, random access memory (RAM) 925, read-only memory (ROM) 920, and hybrids thereof. The storage device 930 may include services 932, 934, and 936 for controlling the processor 910. Other hardware or software modules are intended. The storage device 930 may be connected to the computing device connection 905. In one embodiment, a hardware module performing a particular function may include software components stored in a computer-readable medium that are connected to the necessary hardware components, such as the processor 910, the connection 905, and the output device 935, in order to perform the function.

[0125] The term “computer-readable media” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or transport instructions and / or data. Computer-readable media may also include non-transient media in which data can be stored, and which do not contain carrier waves and / or transient electronic signals that propagate wirelessly or via wired connections. Examples of non-transient media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions, which may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0126] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals, such as bitstreams. However, non-temporary computer-readable storage media, as referred to, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0127] Specific details are provided in the above description to give a complete understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that these embodiments can be practiced without these specific details. For clarity of description, in some cases the art may be presented as including individual functional blocks comprising a device, device components, and software, or steps or routines in a manner embodied in a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, or other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments.

[0128] Individual embodiments may be described above as processes or methods represented as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are complete, but it may have additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or main function.

[0129] The processes and methods described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform a particular function or group of functions, or to otherwise configure a general-purpose computer, a dedicated computer, or a processing device to perform a particular function or group of functions. The portion of the computer resources used may be accessible over a network. Computer-executable instructions may be binary or intermediate format instructions, such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created in the methods described above include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and network-attached storage devices.

[0130] Devices implementing the processes and methods described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. A processor may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functionalities described herein may also be embodied in peripheral devices or add-in cards. Such functionalities may also, as a further example, be implemented on circuit boards in different chips or on different processes running within a single device.

[0131] Instructions, a medium for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described herein.

[0132] While the embodiments of this application are described in the above description with reference to their specific embodiments, those skilled in the art will recognize that this application is not limited thereto. Therefore, although exemplary embodiments of this application are described in detail herein, it should be understood that the concepts of the present invention can be embodied and employed in various other ways, and that, unless limited by the prior art, the appended claims are intended to be interpreted as including such variations. The various features and embodiments of the applications described above can be used individually or in combination. Furthermore, embodiments can be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0133] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this specification.

[0134] When components are described as “configured to” perform certain operations, such configurations can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or by any combination thereof.

[0135] The phrase "combined" refers to any component that is physically connected to another component, either directly or indirectly, and / or any component that communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other preferred communication interface).

[0136] The wording of a claim that includes "at least one of" a set and / or "one or more" of a set indicates that one member of a set or multiple members of a set (in any combination) satisfy the claim. For example, the wording of a claim that includes "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the wording of a claim that includes "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, the wording of a claim that states "at least one of A and B" or "at least one of A or B" could mean A, B, or A and B, and could further include items not listed in the set of A and B.

[0137] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps are outlined above in relation to their functionality. Whether such functionality is implemented as hardware or as software will depend on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this application.

[0138] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. When implemented in software, the technique may be at least partially realized by a computer-readable data storage medium comprising program code that, when executed, includes instructions to perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, etc. The technique may, as an addition or alternative, be at least partially implemented by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures, such as propagating signals or waves, which can be accessed, read, and / or executed by a computer.

[0139] The program code may be executed by a processor which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other uniform integrated or discrete logic circuit configurations. Such processors may be configured to perform any of the techniques described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. Accordingly, the term “processor” as used herein may refer to any of the above structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein.

[0140] Examples used to illustrate this disclosure include the following:

[0141] Embodiment 1: An apparatus comprising a memory and one or more processors coupled to the memory, wherein one or more processors acquire input data comprising stripes that logically segment the input data, each stripe of the input data comprising at least one row of data, and processing the row of data in a first stripe of the input data by each layer of a neural network, wherein the row of data is processed sequentially horizontally according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network, and after processing the row of data in the first stripe, each layer of the neural network A device configured to process subsequent rows of data in a first stripe on a row-by-row basis, wherein each subsequent row of data is processed sequentially in a horizontal direction according to a layer-by-layer sequence; generate a first output stripe based on the processing of the rows of data and subsequent rows of data by a neural network; and, after processing the first stripe, process rows of data in a second stripe of input data on a row-by-row basis by each layer of the neural network, wherein each row of data in the second stripe is processed horizontally according to a layer-by-layer sequence; and generate a second output stripe based on the processing of the second stripe.

[0142] Embodiment 2: The apparatus of Embodiment 1, wherein one or more processors process multiple blocks of data within a row sequentially by each layer of a neural network in order to process rows of data in a first stripe sequentially in the horizontal direction, wherein each layer of the neural network is configured to process each preceding block of data in the depth direction before processing subsequent blocks in the horizontal direction.

[0143] Embodiment 3: The apparatus of Embodiment 2, wherein, in order to process subsequent rows, one or more processors sequentially process each of several blocks of data in each subsequent row by each layer of a neural network, wherein each layer of the neural network is configured to process preceding blocks of data in a subsequent row along the depth direction before processing subsequent blocks of data in the subsequent row.

[0144] Embodiment 4: An apparatus according to any one of Embodiments 1 to 3, wherein the neural network includes an inter-pixel neural network and the input data includes pixels associated with an image.

[0145] Embodiment 5: An apparatus according to any one of Embodiments 1 to 4, wherein one or more processors are configured to perform segmentation, in which each stripe includes a portion of the input data, in order to acquire input data.

[0146] Embodiment 6: An apparatus of any embodiment 1 to 5, wherein one or more processors are configured to store in a first memory the output generated by a previous layer of a neural network for one or more blocks in a row of data within a first stripe, and to store in a second memory associated with a particular layer of a neural network one or more lines of data from one or more blocks in a row of data, wherein the one or more lines of data include a portion of the data input of the particular layer of the neural network for the previous stripe-block-row.

[0147] Embodiment 7: The apparatus of Embodiment 6, wherein one or more processors are configured to generate inputs to a particular layer of a neural network based on combinations of one or more lines of data, including the output generated by a previous layer and a portion of the data input of that particular layer to a previous stripe-block-row, and the particular layer of the neural network generates additional outputs based on the output to the previous layer.

[0148] Embodiment 8: The apparatus of Embodiment 7, wherein one or more processors are configured to determine a portion of the input to a particular layer and to store the portion of the input in a third memory associated with the particular layer of the neural network.

[0149] Embodiment 9: The apparatus of Embodiment 8, wherein one or more processors are configured to generate additional inputs to subsequent layers of a neural network based on a combination of additional outputs of a particular layer and portions of the inputs to the subsequent layer from the stripe-block-row of the subsequent layer, and the subsequent layers of the neural network generate a second additional output based on the additional inputs to the subsequent layers.

[0150] Embodiment 10: The apparatus of Embodiment 9, wherein one or more processors are configured to store a second additional output in a fourth memory, wherein the second and third memories have line stores in line store memory, and the first and fourth memories have buffers in scratch memory.

[0151] Embodiment 11: Any apparatus of Embodiments 1 to 10, wherein one or more processors are configured to store in a first memory associated with a particular layer of the neural network the output generated by a previous layer of the neural network for one or more blocks in each subsequent row of data, and to store in a second memory one or more lines of data from one or more blocks in each subsequent row of data, wherein one or more lines of data include one or more portions of one or more data inputs of a particular layer of the neural network.

[0152] Embodiment 12: The apparatus of Embodiment 11, wherein one or more processors are configured to generate inputs to a particular layer of a neural network based on combinations of one or more lines of data including the output generated by a previous layer and one or more portions of the data inputs of one or more layers of a particular layer to a previous stripe-block-row, and the particular layer of the neural network generates additional outputs based on the output from a previous layer and a subset of the inputs from the particular layer to a previous stripe-block-row.

[0153] Embodiment 13: The apparatus of Embodiment 12, wherein one or more processors are configured to determine a portion of the input to a particular layer, store the portion of the input in a third memory associated with a subsequent layer of the neural network, and store additional outputs in a fourth memory, the second and fourth memories comprising memory buffers for storage.

[0154] Embodiment 14: An apparatus according to any one of Embodiments 1 to 13, wherein one or more processors are configured to process rows of data, with each subsequent layer of the neural network processing the output of the preceding layer of the neural network, such that the output corresponds to a row of data.

[0155] Embodiment 15: An apparatus of any embodiment 1 to 14, wherein one or more processors are configured to process subsequent rows of data by processing the output of the preceding layer of the neural network by each subsequent layer of the neural network, such that the output corresponds to a subsequent row of data in a first stripe.

[0156] Embodiment 16: Any of embodiments 1 to 15, wherein the apparatus includes a camera device.

[0157] Embodiment 17: Any of embodiments 1 to 16, wherein the device comprises a mobile device.

[0158] Embodiment 18: A method comprising the steps of acquiring input data having stripes that logically segment the input data, wherein each stripe of the input data contains at least one row of data; processing rows of data in a first stripe of the input data by each layer of a neural network, wherein the rows of data are processed sequentially horizontally according to a layer-by-layer sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network; and after processing rows of data in the first stripe, processing subsequent rows of data in the first stripe by each layer of the neural network A method comprising: processing rows on a row-by-row basis, wherein each subsequent row of data is processed sequentially horizontally according to a layer-by-layer sequence; generating a first output stripe by a neural network based on the processing of rows of data and subsequent rows of data; processing rows of data in a second stripe of input data on a row-by-row basis by each layer of the neural network after processing the first stripe, wherein each row of data in the second stripe is processed horizontally according to a layer-by-layer sequence; and generating a second output stripe based on the processing of the second stripe.

[0159] Embodiment 19: The method of Embodiment 18, wherein the step of processing rows of data in a first stripe sequentially in the horizontal direction is a step of processing multiple blocks of data in a row sequentially by each layer of a neural network, wherein each layer of the neural network processes each preceding block of data in the depth direction before processing subsequent blocks in the horizontal direction.

[0160] Embodiment 20: The method of Embodiment 19, wherein the step of processing subsequent rows of data is a step of sequentially processing each of several blocks of data in each subsequent row by each layer of the neural network, wherein each layer of the neural network processes preceding blocks of data in the subsequent row along the depth direction before processing subsequent blocks of data in the subsequent row.

[0161] Embodiment 21: Any method from Embodiments 18 to 20, wherein the neural network includes an inter-pixel neural network and the input data includes pixels associated with an image.

[0162] Embodiment 22: Any method of Embodiments 18 to 21, wherein the step of acquiring input data includes the step of logically segmenting the input data into stripes, wherein each stripe includes a portion of the input data.

[0163] Embodiment 23: Any method of Embodiments 18 to 22, further comprising the steps of: storing in a first memory the output generated by a previous layer of a neural network for one or more blocks in a row of data in a first stripe; and storing in a second memory associated with a particular layer of a neural network one or more lines of data from one or more blocks in a row of data, wherein one or more lines of data include a portion of the data input of the particular layer of the neural network for a previous stripe-block-row.

[0164] Embodiment 24: The method of Embodiment 23, further comprising the steps of: generating an input to a particular layer of a neural network based on a combination of one or more lines of data including the output generated by a previous layer and a portion of the data input of the particular layer to the previous stripe-block-row; and generating an additional output by the particular layer of the neural network based on the output to the previous layer.

[0165] Embodiment 25: The method of Embodiment 24, further comprising the steps of determining a portion of the input to a particular layer and storing the portion of the input in a third memory associated with the particular layer of the neural network.

[0166] Embodiment 26: The method of Embodiment 25, further comprising the steps of generating an additional input to a subsequent layer of a neural network based on a combination of an additional output of a particular layer and a portion of the input to the subsequent layer from the stripe-block-row of the subsequent layer; and the subsequent layer of the neural network generating a second additional output based on the additional input to the subsequent layer.

[0167] Embodiment 27: The method of Embodiment 26, further comprising the step of storing a second additional output in a fourth memory, wherein the second memory and the third memory have line stores in line store memory, and the first memory and the fourth memory have buffers in scratch memory.

[0168] Embodiment 28: Any method of Embodiments 18 to 27, further comprising the steps of: storing in a first memory associated with a particular layer of the neural network the output generated by a previous layer of the neural network for one or more blocks in each subsequent row of data; and storing in a second memory one or more lines of data from one or more blocks in each subsequent row of data, wherein one or more lines of data include one or more portions of one or more data inputs of a particular layer of the neural network.

[0169] Embodiment 29: The method of Embodiment 28, further comprising the steps of: generating an input to a particular layer of a neural network based on a combination of one or more lines of data including the output generated by a previous layer and one or more portions of one or more data inputs of a particular layer to a previous stripe-block-row; and generating an additional output by the particular layer of the neural network based on the output from a previous layer and a subset of the inputs from the particular layer to a previous stripe-block-row.

[0170] Embodiment 30: The method of Embodiment 29, further comprising the steps of determining a portion of the input to a particular layer, storing the portion of the input in a third memory associated with a subsequent layer of the neural network, and storing additional outputs in a fourth memory, wherein the second and fourth memories comprise memory buffers.

[0171] Embodiment 31: Any method of Embodiments 18 to 30, wherein the step of processing rows of data includes a step of processing the output of a preceding layer of the neural network by each subsequent layer of the neural network, wherein the output corresponds to a row of data.

[0172] Embodiment 32: Any method of Embodiments 18 to 31, wherein the step of processing subsequent rows of data includes a step of processing the output of a preceding layer of the neural network by each subsequent layer of the neural network, wherein the output corresponds to a subsequent row of data in the first stripe.

[0173] Embodiment 33: A non-temporary computer-readable medium storing instructions, wherein when an instruction is executed by one or more processors, the medium causes one or more processors to execute a method according to any of Embodiments 18 to 32.

[0174] Embodiment 34: An apparatus comprising means for carrying out a method according to any one of Embodiments 18 to 32. [Explanation of symbols]

[0175] 100 Computing Systems 102 Sensors 104 storage 106 memory 110 Computing Components 112 Central Processing Unit (CPU) 114 Graphics Processing Units (GPUs) 116 Digital Signal Processor (DSP) 118 Image Signal Processor (ISP) 120 Neural Processing Units (NPUs) 130 Processing Engines 132 Rendering Engines 200 inputs 202 horizontal dimension 204 Vertical dimension 206 depth dimensions 210A Stripe 210B Stripe 212A Stripe Block 212B Stripe Block 216 Overlapping area 220A Stripe Block 220B Stripe Block 222 Overlapping area 230A The diagram on the left, diagram 230B The figure on the right, Figure 300 Data processing methods, processing methods 302 Stripe Block 304 tiles Line 306A Line 306B Row 306N 310 Neural Networks Layers 312-318 400 memory architecture 402 Line Store Memory 404~410 Line Store 420 scratch memory 422 buffers 424 buffers 452 inputs 454 Duplicate data 456 Output 500 Data Processing Architectures 530 Vertical Composer 532 MAC data paths 534 Line Store Update Components 540 output 542 Duplicate data 544 data 545 Duplicate data 546 Duplicate data 548 output 550 horizontally overlapping data 552 block input 560 Horizontal Composer 562 MAC array 570 Write Controller 572 Read Controller 574 Write Controller 576 Read Controller 602 Layer Input 604 Padding 606 input lines 608 First line 610 Line layer output, layer output 612 output lines 700 Funnel-type processing method 702 Input Stripe 704 Intermediate Stripe 706 Intermediate Stripe 708 Output Stripe 710 Central Stripe 712 horizontal overlap 800 processes 900 Computing Device Architectures 905 Connection, Computing Device Connection 910 Processing Unit (CPU or Processor) 912 cache 915 Computing device memory, memory 920 Read-only memory (ROM) 925 Random Access Memory (RAM) 930 Storage Devices 932 Service 1, Service 934 Service 2, Service 935 Output Device 936 Service 3, Service 940 Communication Interface 945 Input Devices

Claims

1. It is a device, Memory and One or more processors coupled to the memory and The system includes, and the one or more processors Acquiring input data comprising stripes that logically segment the input data, wherein each stripe of the input data contains at least one row of data. Each layer of the neural network processes rows of data within a first stripe of the input data, wherein the rows of data are processed horizontally and sequentially according to a layer-based sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network, and an input to a particular layer of the neural network is generated based on a combination of a stored output generated by the preceding layer of the neural network for one or more blocks within the row of data in the first stripe and one or more lines of stored data from the one or more blocks within the row of data in the first stripe, and for processing by the subsequent layers of the neural network, the particular layer of the neural network generates an additional output based on the output generated by the preceding layer. After processing the rows of data in the first stripe, each layer of the neural network processes subsequent rows of data in the first stripe on a row-by-row basis, wherein each subsequent row of data is processed sequentially in the horizontal direction according to the layer-by-layer sequence. The neural network generates a first output stripe based on the processing of the aforementioned row of data and the subsequent row of data, After processing the first stripe, each layer of the neural network processes the rows of data in the second stripe of the input data on a row-by-row basis, wherein each row of data in the second stripe is processed horizontally according to the layer-by-layer sequence. Based on the processing of the second stripe, a second output stripe is generated. A device configured to perform the following actions.

2. To process the rows of data within the first stripe continuously in the horizontal direction, one or more processors: The process involves each layer of the neural network processing multiple blocks of data within a row sequentially, wherein each layer of the neural network processes each preceding block of data along the depth direction before processing subsequent blocks along the horizontal direction. The apparatus according to claim 1, configured to perform the following:

3. In order to process the subsequent rows, one or more processors Each layer of the neural network processes multiple blocks of data in each subsequent row sequentially, wherein each layer of the neural network processes the preceding blocks of data in the subsequent row along the depth direction before processing the subsequent blocks of data in the subsequent row. The apparatus according to claim 2, configured to perform the following:

4. The aforementioned neural network includes an inter-pixel neural network, The apparatus according to claim 1, wherein the input data includes pixels associated with an image.

5. In order to acquire the aforementioned input data, one or more processors The input data is logically segmented into stripes, wherein each stripe includes a portion of the input data. The apparatus according to claim 1, configured to perform the following:

6. The one or more processors described above The output generated by the previous layer of the neural network for one or more blocks in the row of the data in the first stripe is stored in the first memory, Storing one or more lines of data from one or more blocks within the row of data in the first stripe in a second memory associated with the particular layer of the neural network, wherein the one or more lines of data include the portion of the data input of the particular layer of the neural network to the previous row of data. The apparatus according to claim 1, configured to perform the following:

7. The one or more processors described above Determining the portion of the input for the specific layer, The portion of the input is stored in a third memory associated with the specific layer of the neural network. The apparatus according to claim 6, configured to perform the following:

8. The one or more processors described above To generate additional inputs to the subsequent layer of the neural network based on the combination of the additional output of the particular layer and the portion of the input to the subsequent layer from the previous row of data of the subsequent layer, The subsequent layer of the neural network generates a second additional output based on the additional input to the subsequent layer. It is configured to do the following: The one or more processors described above The second additional output is stored in a fourth memory, wherein the second memory and the third memory have a line store in the line store memory, and the first memory and the fourth memory have a buffer in the scratch memory. The apparatus according to claim 7, configured to perform the following:

9. The one or more processors described above The output generated by the previous layer of the neural network for one or more blocks in each subsequent row of data is stored in a first memory associated with a particular layer of the neural network. The storage of one or more lines of data from one or more blocks in each subsequent row of data in a second memory, wherein the one or more lines of data include one or more portions of one or more data inputs of one or more layers of the particular layer of the neural network. The apparatus according to claim 1, configured to perform the following:

10. The one or more processors described above The input to the particular layer of the neural network is generated based on the combination of the output generated by the previous layer and the one or more lines of data that include the one or more portions of the one or more data inputs of the particular layer for the previous row of data, The particular layer of the neural network generates an additional output based on the output from the previous layer and a subset of the input from the particular layer to the previous row of data. It is configured to do the following: The one or more processors described above Determining the portion of the input for the specific layer, The portion of the input is stored in a third memory associated with a subsequent layer of the neural network. The additional output is stored in a fourth memory, wherein the second memory and the fourth memory are equipped with memory buffers for storage. The apparatus according to claim 9, configured to perform the following:

11. To process the aforementioned row of data, one or more processors, Processing the output of the preceding layer of the neural network by each subsequent layer of the neural network, wherein the output corresponds to the row of data. The apparatus according to claim 1, configured to perform the following:

12. To process the subsequent rows of data, one or more processors, Processing the output of the preceding layer of the neural network by each subsequent layer of the neural network, wherein the output corresponds to the subsequent row of data in the first stripe. The apparatus according to claim 1, configured to perform the following:

13. The apparatus according to claim 1, wherein the apparatus comprises a camera device and / or a mobile device.

14. It is a method, A step of acquiring input data comprising stripes that logically segment the input data, wherein each stripe of the input data includes at least one row of data; A step of processing rows of data in a first stripe of input data by each layer of a neural network, wherein the rows of data are processed horizontally sequentially according to a layer-based sequence in which each preceding layer of the neural network processes a particular row of data before each subsequent layer of the neural network, and an input to a particular layer of the neural network is generated based on a combination of a stored output generated by the preceding layer of the neural network for one or more blocks in the row of data in the first stripe and one or more lines of stored data from the one or more blocks in the row of data in the first stripe, and for processing by the subsequent layer of the neural network, the particular layer of the neural network generates an additional output based on the output generated by the preceding layer. A step of processing the rows of data in the first stripe, and then processing subsequent rows of data in the first stripe on a row-by-row basis using each layer of the neural network, wherein each subsequent row of data is processed sequentially in the horizontal direction according to the layer-by-layer sequence, The neural network generates a first output stripe based on the processing of the row of data and the subsequent row of data. After processing the first stripe, the neural network processes each layer of the input data row by row, wherein each row of data in the second stripe is processed horizontally according to the layer-based sequence. A step of generating a second output stripe based on the processing of the second stripe, and Methods that include...

15. A non-temporary computer-readable recording medium storing instructions, wherein when an instruction is executed by one or more processors, the recording medium causes the one or more processors to execute the method described in claim 14.

Images