Efficient demosaicing on neural processing units

Abstract

Techniques and systems for image demosaicing are provided. For example, a process may include performing a deep convolution operation on image data and a deep convolution filter having predetermined parameter values to obtain a plurality of color channels of the image data; performing a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arranging the processed plurality of color channels into a demosaiced image; and outputting the demosaiced image.

Description

TECHNICAL FIELD

[0001] This application relates to image processing. For example, aspects of the application relate to systems and techniques for efficient demosaicking on processing units (e.g., neural processing units (NPUs)).

[0002] BACKGROUND

[0003] Many devices and systems allow for capturing a scene by generating image (or frame) and / or video data (including multiple frames). For example, a camera or a device including a camera (or multiple cameras) can capture a sequence of frames of a scene (e.g., a video of a scene) based on light entering the camera. To improve the quality of the frames captured by the camera, the camera can include a lens to focus the light entering the camera. The sequence of frames captured by the camera can be output for display, for processing and / or consumption by other devices, among other uses.

[0004] A camera can typically include one or more processors, such as an image signal processor (ISP), which can process one or more image frames captured by an image sensor. For example, a raw image frame captured by an image sensor can be processed by an image signal processor (ISP) to generate a final image. A camera can be configured with various image capture and image processing settings to change the appearance of an image. Some camera settings are determined and applied prior to or during capturing an image, such as ISO, exposure time (also referred to as exposure duration), aperture size, f / stop, shutter speed, focus, and gain, among others. Increasingly, image processing tasks performed prior to or concurrently with image capture are being performed using artificial intelligence (AI) / machine learning (ML) models. SUMMARY

[0005] Systems and techniques for image processing are described herein. The following presents a simplified summary of one or more aspects related to the mechanisms disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the summary be considered to identify key or critical elements relating to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary is presented in a simplified form to present some concepts related to one or more aspects relating to the mechanisms disclosed herein before presenting a more detailed description in the following section.

[0006] Systems, apparatuses, methods, and computer-readable media are provided that disclose image processing. In one illustrative example, an apparatus for image demosaicing is provided. The apparatus includes at least one memory configured to store image data; and at least one processor coupled to the at least one memory. The at least one processor is configured to perform a depthwise convolution operation on the image data and a depthwise convolution filter having predetermined parameter values to obtain a plurality of color channels of the image data; perform a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arrange the plurality of processed color channels into a demosaiced image; and output the demosaiced image.

[0007] In another example, a non-transitory computer-readable medium including instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform a depthwise convolution operation on image data and a depthwise convolution filter having predetermined parameter values to obtain a plurality of color channels of the image data; perform a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arrange the plurality of processed color channels into a demosaiced image; and output the demosaiced image

[0008] In another example, a non-transitory computer-readable medium including instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform a depthwise convolution operation on image data and a depthwise convolution filter having predetermined parameter values to obtain a plurality of color channels of the image data; perform a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arrange the plurality of processed color channels into a demosaiced image; and output the demosaiced image

[0009] In another example, a non-transitory computer-readable medium including instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform a depthwise convolution operation on image data and a depthwise convolution filter having predetermined parameter values to obtain a plurality of color channels of the image data; perform a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arrange the plurality of processed color channels into a demosaiced image; and output the demosaiced image

[0010] In some aspects, one or more of the devices described herein can include or be part of an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device (e.g., a network-connected watch or other wearable device), a personal computer, a laptop computer, a server computer, a television, a video game console, or other device. In some aspects, the device further includes at least one camera for capturing one or more images or video frames. For example, the device can include one camera (e.g., an RGB camera) or multiple cameras for capturing one or more images and / or one or more videos including video frames. In some aspects, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the device includes a transmitter configured to transmit data or information to at least one device over a transmission medium. In some aspects, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or other processing device or component.

[0011] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood from the entire specification and any or all embodiments described herein.

[0012] The foregoing will be more readily understood when considered in reference to the following description, when viewed in association with the accompanying drawings, in which: BRIEF DESCRIPTION OF DRAWINGS

[0013] Exemplary examples of the present application are described in detail below with reference to the following drawings: Figure 1 is a block diagram illustrating an architecture of an image capture and processing system in accordance with aspects of the present disclosure.

[0014] Figure 2A is a diagram illustrating an example of a fully connected neural network in accordance with some examples of the present disclosure; Figure 2B is a diagram illustrating an example of a locally connected neural network in accordance with some examples of the present disclosure; Figure 2C is a diagram illustrating an example of a convolutional neural network in accordance with some examples of the present disclosure; Figure 2D is a diagram illustrating an example of a deep convolutional network (DCN) for identifying visual features from images in accordance with some examples of the present disclosure; Figure 3is a block diagram illustrating an example deep convolutional network (DCN) in accordance with some examples of the present disclosure; Figure 4 is a block diagram illustrating an image processing pipeline for processing an image in accordance with aspects of the present disclosure; Figure 5 is a computational graph illustrating a technique 500 for efficient demosaicking on an NPU in accordance with aspects of the present disclosure; Figure 6 is a flowchart illustrating a process for demosaicking image data in accordance with aspects of the present disclosure; Figure 7 example implementations of a system on chip (SoC) in accordance with aspects of the present disclosure are illustrated; and Figure 8 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. DETAILED DESCRIPTION

[0015] Certain aspects and examples of the present disclosure are provided below. As will be apparent to those of ordinary skill in the art, some of these aspects and examples can be employed independently, and some of these aspects and examples can be employed in combination with others. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of the subject matter of the application. However, it will be apparent that various examples can be practiced without these specific details. The drawings and description are not intended to be limiting.

[0016] The following description provides exemplary examples only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the following description will provide those skilled in the art with an enabling description for implementing exemplary examples. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0017] In some cases, a machine learning (ML) model, such as a convolutional neural network (CNN), can be used to process raw image data. Raw image data can be unprocessed data captured by an image sensor. In some cases, the raw image data is captured using a color filter array, such as a Bayer filter (e.g., a Bayer filter). For a grid of four pixels (e.g., a 2x2 matrix), raw image data captured by an image sensor using a Bayer filter can have two pixels with green data pixel values, one pixel with red data pixel values, and one pixel with blue data pixel values. In some cases, the raw image data can be processed at least in part by demosaicking. Demosaicking reconstructs a full color image from the raw image data such that each pixel includes red, green, and blue data. In some cases, a ML system can be used to perform demosaicking.

[0018] In some cases, an ML system can be executed on a neural processing unit (NPU). In some cases, the ML system can perform convolution operations, and the NPU can include multiply and accumulate (MAC) units to accelerate the execution of the convolution operations. However, existing ML-based demosaicing techniques tend to include a large number of memory-intensive operations and relatively few convolution operations, which can inefficiently utilize the MAC units of the NPU and waste NPU cycles when executed on certain NPUs. Since NPUs are generally not optimized for performing memory-intensive operations, existing ML-based demosaicing techniques tend to run relatively poorly on NPUs as compared to CPUs / GPUs. In some cases, it can be useful to provide a technique for efficient demosaicing on an NPU using an ML algorithm.

[0019] Systems, apparatuses, electronic devices, methods (also referred to as processes), and computer-readable media (herein collectively referred to as “systems and techniques”) for efficient demosaicing on an NPU are described herein. For example, image data, such as raw image data, can be received and a deep convolution operation can be performed on the image data. The deep convolution operation can use a deep convolution filter having predetermined parameter values. The predetermined parameter values can be predetermined based on a color filter of an image sensor used to capture the image data. The predetermined parameter values can be determined to extract pixel color values from the image. The deep convolution operation can extract the pixel color values to produce a set of stitched color channels, such as a red channel, a blue channel, and a green channel. The color channels can be passed to a convolution operation block. The convolution operation block can use one or more convolution layers to perform an interpolation operation on the stitched color channels. The interpolated color channels can then be arranged into color channels of a processed image for output by a depth-to-space block. In some cases, the deep convolution filter operation, the convolution operation, and the depth-to-space operation can be executed on MAC units of an NPU.

[0020] Various aspects of the present application will be described with respect to the following figures.

[0021] Figure 1is a block diagram illustrating an architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., images of the scene 110). The image capture and processing system 100 can capture individual images (or photos), and / or can capture video including multiple images (or video frames) in a particular sequence. In some cases, the lens 115 and the image sensor 130 can be associated with an optical axis. In one illustrative example, both the light sensitive area (e.g., photodiodes) of the image sensor 130 and the lens 115 can be centered on the optical axis. The lens 115 of the image capture and processing system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the incoming light from the scene toward the image sensor 130. The light received by the lens 115 passes through an aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and received by the image sensor 130. In some cases, the aperture can have a fixed size.

[0022] The one or more control mechanisms 120 can control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 can include multiple mechanisms and components; for example, the control mechanisms 120 can include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 can also include additional control mechanisms beyond those illustrated, such as control mechanisms that control analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0023] The focus control mechanism 125B of the control mechanism 120 can obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust a positioning of the lens 115 relative to a positioning of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can cause the lens 115 to move closer to or further away from the image sensor 130 by actuating a motor or servo system (or other lens mechanism), thereby adjusting the focus. In some cases, an additional lens can be included in the image capture and processing system 100, such as one or more micro-lenses on each photodiode of the image sensor 130 that each bend light received from the lens 115 toward the corresponding photodiode before the light reaches the photodiode. The focus setting can be determined via contrast-detect autofocus (CDAF), phase-detect autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting can be referred to as an image capture setting and / or an image processing setting. In some cases, the lens 115 can be fixed relative to the image sensor, and the focus control mechanism 125B can be omitted without departing from the scope of the present disclosure.

[0024] The exposure control mechanism 125A of the control mechanism 120 can obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 125A can control a size of an aperture (e.g., an aperture size or an aperture value), a time duration over which the aperture is open (e.g., an exposure time or a shutter speed), a time duration over which the sensor collects light (e.g., an exposure time or an electronic shutter speed), a sensitivity of the image sensor 130 (e.g., an ISO speed or a film speed), an analog gain applied by the image sensor 130, or any combination thereof. The exposure setting can be referred to as an image capture setting and / or an image processing setting.

[0025] The zoom control mechanism 125C of the control mechanism 120 can obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control a focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by causing one or more motors or servo systems (or other lens mechanisms) to actuate to move one or more of the lenses relative to each other. The zoom setting can be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly can include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly can include a focusing lens (which in some cases can be the lens 115) that first receives light from the scene 110, where the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, the afocal zoom system can include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other) with a negative (e.g., diverging, concave) lens between the two positive lenses. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses. In some cases, the zoom control mechanism 125C can control zooming by capturing an image from an image sensor (e.g., including the image sensor 130) of a plurality of image sensors at a zoom corresponding to the zoom setting. For example, the image processing system 100 can include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, the zoom control mechanism 125C can capture an image from the corresponding sensor.

[0026] Image sensor 130 includes one or more arrays of photodiodes or other light- sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in an image produced by image sensor 130. In some cases, different photodiodes can be covered by different filters. In some cases, different photodiodes can be covered in different color filters, and can thus measure light that matches a color of the filter covering the photodiode. Various color filter arrays can be used, including a Bayer color filter array, a four-color filter array (also referred to as a four-Bayer color filter array or QCFA), cyan, yellow, green, magenta (CYGM filter), and / or any other color filter array. For example, a Bayer color filter includes red color filters, blue color filters, and green color filters, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red color filter, blue light data from at least one photodiode covered in a blue color filter, and green light data from at least one photodiode covered in a green color filter.

[0027] Returning to Figure 1 , other types of color filters can use yellow, magenta, and / or cyan (also referred to as “emeraldine”) color filters as an alternative or in addition to red, blue, and / or green color filters. In some cases, some photodiodes can be configured to measure infrared (IR) light. In some implementations, photodiodes that measure IR light can not be covered by any filter, thus allowing the IR photodiodes to measure both visible light (e.g., color) and IR light. In some examples, IR photodiodes can be covered by an IR filter, thus allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 130) can lack filters (e.g., color, IR, or any other part of the spectrum) entirely, and can instead use different photodiodes (in some cases, vertically stacked) throughout the pixel array. The different photodiodes throughout the pixel array can have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors can also lack filters, and thus lack color depth.

[0028] In some cases, image sensor 130 may optionally or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or the analog signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms in control mechanism 120 may be alternatively or additionally included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron multiplication CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0029] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), neural processing unit (NPU) 158, and / or related to Figure 7The computing system 700 may include one or more processors 710 of any other type discussed herein. The host processor 152 may be a digital signal processor (DSP) and / or other types of processor. In some specific implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) that includes the host processor 152, NPU 158, and ISP 154. Among other circuitry, the NPU may also include input buffers, weight buffers, a set of multiply and accumulate units, and an output buffer (not shown), and the NPU may be optimized to perform convolutional computations common in ML models such as convolutional neural networks, deep convolutional networks, etc. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer port or interface, Advanced High Performance Bus (AHB) bus, any combination thereof and / or other input / output ports). In an exemplary example, host processor 152 may use the I2C port to communicate with image sensor 130, and ISP 154 may use the MIPI port to communicate with image sensor 130.

[0030] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1025, read-only memory (ROM) 145 / 1020, cache, memory unit, another storage device, or some combination thereof.

[0031] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, touchpad, touch-sensitive surface, printer, any other output device, any other input device, or some combination thereof. In some cases, text may be input into the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad on the touchscreen of the I / O device 160. I / O devices 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 can receive data from and / or send data to one or more peripheral devices. I / O devices 160 may include one or more wireless transceivers that enable a wireless connection between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 can receive data from and / or send data to one or more peripheral devices. Peripheral devices may include any type of I / O device 160 discussed earlier, and they can be considered I / O devices 160 in themselves once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors.

[0032] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may be two or more independent devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0033] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154, a host processor 152, and an NPU 158), RAM 140, ROM 145, and I / O devices 160. In some cases, certain components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0034] Image capture and processing system 100 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.10 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some specific implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0035] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown herein are additional components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0036] In some cases, images captured by the image capture and processing system 100 can be processed by a neural network and / or a machine learning (ML) system. A neural network is an example of an ML system, and it may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is produced by output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may include one or more weights used to indicate the importance of nodes in one or more layers. In some cases, a deep learning network may have a series of many hidden layers, where earlier layers are used to determine simple and low-level features of the input, while later layers build a hierarchy of more complex and abstract features.

[0037] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0038] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0039] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input. Connections between layers of a neural network can be fully connected or locally connected. Further details are discussed below.Figure 2A to Figure 3 Describe various examples of neural network architectures.

[0040] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0041] The connections between layers in a neural network can be fully connected or locally connected. Figure 2A An example of a fully connected neural network 202 is illustrated. In the fully connected neural network 202, neurons in the first layer can transmit their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 2B An example of a locally connected neural network 204 is illustrated. In the locally connected neural network 204, neurons in the first layer can connect to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 210, 212, 214, and 216). The connectivity pattern of local connections may produce spatially different receptive fields in higher layers because higher-layer neurons in a given region can receive inputs that are trained by properties tuned to a restricted portion of the network's total input.

[0042] An example of a locally connected neural network is a convolutional neural network. Figure 2C An example of a convolutional neural network 206 is illustrated. The convolutional neural network 206 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful. According to various aspects of this disclosure, the convolutional neural network 206 can be used to perform one or more aspects of video compression and / or decompression.

[0043] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2DA detailed example of DCN 200 is illustrated. DCN is designed to receive signals from image capture device 230 (such as...). Figure 1 The image capture and processing system 100 identifies visual features from the input image 226. The DCN 200 in the current example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 200 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0044] Supervised learning can be used to train the DCN 200. During training, an image (such as image 226 of a speed limit sign) can be presented to the DCN 200, and forward passes can then be computed to produce output 222. The DCN 200 may include a feature extraction part and a classification part. Upon receiving image 226, convolutional layer 232 can apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. As an example, the convolutional kernel used for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 218, four different convolutional kernels are applied to image 226 at convolutional layer 232. Convolutional kernels may also be referred to as filters or convolutional filters.

[0045] The first set of feature maps 218 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 220. The max-pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220 (e.g., 14x14) is smaller than the size of the first set of feature maps 218 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0046] exist Figure 2D In the example, the second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as "sign", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of the DCN 200 is the probability that the image 226 includes one or more features.

[0047] In this example, the probabilities for "sign" and "60" in output 222 are higher than the probabilities for other numbers in output 222 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 222 generated by DCN 200 may be incorrect. Therefore, the error between output 222 and the target output can be calculated. The target output is the ground truth value of image 226 (e.g., "sign" and "60"). The weights of DCN 200 can then be adjusted so that output 222 of DCN 200 is more closely aligned with the target output.

[0048] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates by how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0049] In practice, the error gradient of the weights can be computed in a small number of examples to make the computed gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches the target level. After learning, a new image can be presented to the DCN, and the forward pass of the network can produce an output that can be considered an inference or prediction of the DCN.222

[0050] Deep convolutional networks (DCNs) are convolutional networks configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0051] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0052] The processing at each layer of a convolutional network can be considered as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be considered three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The output of the convolutional connections can be considered as forming feature maps in subsequent layers, where each element in the feature map (e.g., feature map 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature maps can be further processed with nonlinearities (such as corrections, maximum values ​​(0, x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction.

[0053] Figure 3 This is a block diagram illustrating an example of a deep convolutional network 350. Based on connectivity and weight sharing, the deep convolutional network 350 can include multiple layers of different types. For example... Figure 3 As shown, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each convolutional block in 354A and 354B can be configured with a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360. It is worth noting that the layers illustrated with respect to convolutional blocks 354A and 354B are examples of layers that can be included in convolutional layers and are not intended to be limiting, and other types of layers (and additional numbers of layers) can be included in any order.

[0054] Convolutional layer 356 may include one or more convolutional filters that can be applied to input data 352 to generate feature maps. Although only two convolutional blocks 354A and 354B are shown, this disclosure is not limited thereto, and any number of convolutional blocks (e.g., convolutional blocks 354A and 354B) may be included in the deep convolutional network 350 according to design preferences. Normalization layer 358 may normalize the output of the convolutional filters. For example, normalization layer 358 may provide whitening or lateral suppression. Max pooling layer 360 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0055] For example, the parallel filter bank of the deep convolutional network can be loaded onto a processor such as a CPU or GPU, or any other type of processor 1010 discussed with respect to the computing system 1000 of Figure 10, to achieve high performance and low power consumption. Alternatively, the parallel filter bank can be loaded onto the DSP or ISP of the computing system 1000 of Figure 10. Furthermore, the deep convolutional network 350 can access other processing blocks that may exist on the computing system 1000 of Figure 10, such as sensor processors and navigation modules dedicated to sensors and navigation, respectively.

[0056] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may also include a logistic regression (LR) layer 364. Weights (not shown) to be updated between each layer 356, 358, 360, 362A, 362B, 364 of the deep convolutional network 350. The output of each of these layers (e.g., 356, 358, 360, 362A, 362B, 364) can serve as the input to the next layer in the deep convolutional network 350 (e.g., 356, 358, 360, 362A, 362B, 364) to learn hierarchical feature representations from the input data 352 (e.g., images, audio, video, sensor data, and / or other input data) provided at the initial convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 of the input data 352. The classification score 366 can be a set of probabilities, where each probability is the probability that the input data includes a feature from the feature set.

[0057] In some cases, one or more convolutional networks (such as DCNs) can be incorporated into a more complex ML network. As an example, as indicated above, a deep convolutional network 350 can output the probability that input data (such as an image) includes certain features. The deep convolutional network 350 can then be modified to extract (e.g., output) certain features. Additionally, DCNs can be added to extract other features. This set of DCNs can be used as a feature extractor to identify features in an image. In some cases, feature extractors can be used as the backbone of additional ML network components to perform further operations.

[0058] In some cases, ML networks can be applied to raw image data. Raw image data can be unprocessed data captured by an image sensor. In digital imaging, color filter arrays, such as Bayer color filters (e.g., Bayer filters), are typically used to capture raw image data. A color filter array consists of color filters placed above the pixels of an image sensor, such that each pixel detects a wavelength corresponding to a certain color and provides image data for that color. The color filters can be arranged in a pixel pattern, such as a Bayer array with one blue pixel, one red pixel, and two green pixels arranged in a 2×2 pixel matrix. Therefore, in the raw image data of an image sensor using a Bayer filter, for a four-pixel grid (e.g., a 2×2 matrix), two pixels will include only green data, one pixel will include only red data, and one pixel will include only blue data.

[0059] In many cases, raw image data can be processed to generate a processed image for output. As part of this processing, demosaicing can be performed on the raw image data. Demosaicing reconstructs a full-color image from the raw image data, ensuring that each pixel includes red, green, and blue data. In some cases, demosaicing can be performed by an image signal processor (ISP) coupled to the image sensor. The ISP can also perform various other processing functions to improve the raw image data, such as noise reduction, white balance, and contrast enhancement. This additional processing can be performed before or after demosaicing. In some cases, an ML system can be used to perform demosaicing and / or other additional processing that can be performed on the raw image data.

[0060] Figure 4 This is a block diagram illustrating an image processing pipeline 400 for processing images according to various aspects of the present disclosure. In the image processing pipeline 400, image data 402, for example, from an image sensor, may be input to an ISP 404. In this example, image data 402 is raw image data of a Bayer array format 408. In other cases, the ISP 404 may process other types of image data, such as image data obtained using different color filters. In some cases, the ISP 404 may be configured to process image data 402 using a conventional Bayer enhancement and demosaic pipeline 410 (e.g., using black level adjustment blocks, noise reduction blocks, white balance blocks, interpolation, etc.). In other cases, the ISP 404 may be configured to process image data 402 using an ML-based Bayer enhancement and demosaic pipeline 412. In this example, the ML-based Bayer enhancement and demosaic pipeline 412 includes one or more ML models for Bayer enhancement 414 and an ML model for demosaic 416. In some cases, one or more ML models from the Bayer Enhancement and Demosaic Pipeline 412 can be executed on the NPU 406. The output from the conventional Bayer Enhancement and Demosaic Pipeline 410 and / or the ML-based Bayer Enhancement and Demosaic Pipeline 412 can be further processed to convert images from RGB to YCC format, enhance edges and contrast, etc.

[0061] Efficiently performing demosaicing on an NPU can be challenging because common demosaicing operations involve extracting (e.g., separating) red, green, and blue pixels from an input Bayer array image into separate images, padding and interpolating to fill in blank pixels in the extracted images, and then stitching the images together into a single output image with the red, green, and blue pixel values ​​for each pixel. In some cases, when applied to relatively large images such as 4K+ images, existing ML-based demosaicing techniques may perform poorly when executed on the NPU 406. For example, the NPU 406 may include a relatively large number of multiply-accumulate (MAC) hardware units optimized to accelerate convolution-based operations. Such operations compute the product of two values ​​(e.g., numbers, vectors, matrices, etc.) and store the result in an accumulator (such as a register), which can then be fed back via the MAC. However, existing ML-based demosaicing techniques involve relatively few operations utilizing the MAC hardware units and a large number of memory-intensive operations such as extraction / slicing, padding, and stitching operations. Many NPUs may not be optimized for memory-intensive operations, such as extracting portions of data in non-linear patterns (e.g., certain color information), padding data (e.g., interpolation), and splicing data, where data in the NPU 406's accumulators / registers / cache / on-chip memory, etc., is heavily written to and / or read from. Therefore, in many cases, existing ML-based techniques for demosaicing may perform better on a CPU or GPU 420 than on the NPU 406, as the CPU / GPU 420 is better optimized for memory-intensive operations, even if it may not perform well for MAC operations. In some cases, ML-based demosaicing techniques for minimizing memory-intensive operations may be useful to improve the performance of ML-based demosaicing on the NPU 406's MAC hardware unit.

[0062] Figure 5This is a computational diagram illustrating a technique 500 for efficient demosaicing on an NPU according to various aspects of this disclosure. In this example, the ML model 520 for demosaicing may include a depth convolution block 504, a convolution block 506, and a depth-to-space block 508. In technique 500, a Bayer array input image 502 may be fed into the depth convolution block 504 of the ML model 520 for demosaicing to obtain a set of color channel images (e.g., an image including only red values, an image including only green values, and an image including only blue values). It is worth noting that although discussed in the context of the Bayer array input image 502, it is understood that technique 500 can be applied to any patterned input image. In some cases, the input image 502 may be in a single channel with one color value per pixel. The depth convolution block 504 may perform depth convolution on one or more portions of the input image 502. In the case of using a single portion and the input image 502 being a Bayer array image, three color channels may be output: one red, one green, and one blue color channel.

[0063] In some cases, the input image can be divided into multiple parts. Each part can be input as a separate channel to the ML model 520 for demosaicing. For example, the image can be divided into eight (8) parts and input as eight batches (e.g., eight channels) to the deep convolutional block 504. Each batch can run concurrently and independently of other batches (e.g., each batch can be considered as a separate image). For understanding and clarity, the deep convolutional block 504 will be discussed in the context of a single batch, and the concepts and techniques discussed about that batch can be applied to other batches.

[0064] In some cases, the depthwise convolutional block 504 may take input, such as input image 502, and perform a separate convolution operation with each filter in a set of filters to generate a separate output channel corresponding to each filter. For example, using a 2x2 kernel size and a stride of 2, a 2x2 pixel portion of the input image may be selected, and the selected pixel portion may be convolved with each filter in the set of filters. In some cases, the filters in the set of filters may be predetermined. The filters of this depthwise convolution are fixed and not trainable (e.g., the filters are predefined and do not change during the training of the ML model). In some cases, the depthwise convolutional block 504 may be replaced by one or more convolutional blocks configured to perform depthwise convolution (e.g., convolving the pixel values ​​of input image 502 with each filter and producing a color channel corresponding to each filter). In some cases, depthwise convolution may be performed using a single software instruction, while the convolution-based version may be performed using multiple (or nested) software instructions.

[0065] A set of filter values ​​(e.g., predetermined parameter values) can be determined to allow the NPU's MAC unit to extract pixel color values ​​from the input image 502, and the filter values ​​can be based on the pixel pattern of the input image. For example, for a 2x2 pixel portion of a Bayer array image, the top-left pixel could be a blue pixel value 522A, the top-right pixel could be a green pixel value 522B, the bottom-left pixel could be a green pixel value 522C, and the bottom-right pixel could be a red pixel value 522D. For this type of 2x2 pixel portion, the first filter 526 can be [[[[1.0]], [[0.0]]], [[[0.0]], [[0.0]]]], the second filter 528 can be [[[[0.0]], [[0.0]]], [[[0.0]], [[1.0]]]], and the third filter 530 can be [[[[0.0]], [[1.0]]], [[[1.0]], [[0.0]]]]. When applied to the 2x2 pixel portion, the first filter multiplies the blue pixel value by 1 and multiplies all other color pixel values ​​by 0, leaving the blue pixel value and zeroing out the other color values ​​(e.g., effectively removing them) to output on the blue channel. Applying the second filter to the 2x2 pixel portion multiplies the green pixel value by 1 and zeroes out the other color pixel values, leaving the green pixel value to output on the green channel. Similarly, applying a third filter to a 2x2 pixel portion multiplies the red pixel value by 1 and zeroes out the other color pixel values, leaving the red pixel value for output on the red channel. In some cases, the values ​​of the filters to be applied can be determined based on (e.g., the pixel pattern expected to be present in the input image data of the image sensor, such as BGGR, RGGB, etc.), and these values ​​can be predefined according to the expected pixel pattern. In some cases, the filter values ​​can be determined based on the expected pixel pattern of the image sensor used to capture the image data. In some cases, for example, if the image sensor is configured to capture image data with different pixel patterns based on the image capture mode and / or settings, multiple filter values ​​can be determined for the image sensor.

[0066] In some cases, depthwise convolution of a 2x2 pixel portion can be performed on the NPU's MAC unit. For example, the filter can be stored in the NPU's weight buffer, and the pixel portion can be stored in the NPU's input buffer. Both the filter and the pixel portion can be fed into the MAC for processing as a simple multiplication operation, and the output is stored on the MAC's accumulator in a separate channel. The depthwise convolution block 504 and the NPU can then be used in a substantially similar manner to process the next 2x2 pixel portion of the input image 502 (e.g., with a stride of 2). In some cases, the application of the filter to the pixel portion can be performed within one cycle of the NPU. In some cases, using 1 and 0 as the only values ​​in the filter can help allow the NPU's MAC unit to extract the corresponding color value from the 2x2 pixel portion without activating the entire MAC circuitry, as certain patterns of discrete zeros and one can help reduce the power consumption of some NPU multiplication and memory operations. The exact pattern of discrete zeros and one can be hardware-dependent. Additionally, the 0-to-1 filter weights are kept within a 1-bit quantization range and are therefore favorable for quantization, which helps reduce the size of the ML model and thus reduces the energy required to execute the model at a given accuracy level. In some cases, the pixel portions (e.g., kernels) can be of any size and multiple pixel portions can be processed in parallel. The output of the depthwise convolutional block 504 can be a concatenated color channel for each input color (e.g., an image). Thus, for RGB, three color channels 524 can be output. In some cases, the resolution of the concatenated color channels can be H / 2 × W / 2 for the input resolution (H × W) of the input image. It is worth noting that while NPUs including MAC units are discussed, it should be understood that the techniques discussed herein are applicable to any processor that includes a MAC unit. For example, CPUs and / or GPUs may include MAC units and the techniques discussed herein are applicable to such CPUs and / or GPUs. As another example, an NPU can be implemented as an ASIC that includes a MAC unit for executing the techniques discussed herein and an ML model. Additionally, in some examples, the NPU can be integrated into an image sensor.

[0067] When the input image 502 is divided into multiple batches, each batch can output a set of stitched color channels based on the pixel pattern, since each batch can run simultaneously and independently of other batches. For example, for a Bayer array input image with 8 batches, each batch outputs 3 stitched color channels.

[0068] The color channels stitched from the output of depth convolution block 504 can be input to convolution block 506. Convolution block 506 may include one or more ML layers performing one or more convolution operations. Convolution block 506 may perform interpolation operations on the stitched color channels output by depth convolution block 504. In some cases, convolution block 506 may also perform other operations, such as color correction, white balance, edge sharpening, etc. In some cases, convolution block 506 may interpolate the color channel input from the output of depth convolution block 504 to output a color channel 534 with the same size as the input image 502. The output color channel 534 may therefore have the same resolution as the input image 502 (shown as H×W).

[0069] In some respects, convolutional block 506 can use 3x3 accept attention across channels (e.g., across color channels, such as red, green, and blue channels) to predict (e.g., interpolate) additional color values. In some cases, convolutional block 506 can be configured to output multiple channels per input channel. In some cases, the number of output channels per input concatenated color channel can be based on how much amplification is to be performed. In some cases, the number of filters and output channels of convolutional block 506 can be determined as a high amplification factor. Width magnification factor The desired number of output channels is a function of the desired number of output channels. For example, as discussed above, the resolution of the stitched color channels could be H / 2 × W / 2. In such an example, the height magnification factor could be 2 and the width magnification factor could also be 2 (e.g., to provide an output with the original resolution H × W), and the desired number of output channels in the output image is 3 (e.g., R, G, B), so convolutional block 506 can output 2 2 3 = 12 channels (shown as channel 532). In some cases, convolutional block 506 can be configured to generate a 12-channel output by configuring the convolution operation of convolutional block 506 with 12 convolutional filters having trainable weights.

[0070] Convolutional block 506 can be trainable to perform interpolation operations. As indicated above, depth convolutional block 504 may be non-trainable. Additionally, depth-to-space block 508 may also be non-trainable. Convolutional block 506 can be trainable to perform interpolation operations. In some cases, the dataset used to train the ML model 520 for demosaicing can be obtained by acquiring a large number of images in an output format (such as RGB) and processing the images to convert them into an input format (such as an image with a Bayer array). For example, this processing can be performed by removing each pixel based on the input format (e.g., Bayer array, quad Bayer, CMYW, etc.) until only one color remains (e.g., leaving only one of the red, green, or blue values ​​per pixel). For example, an image can be converted to a Bayer array image. The Bayer array image and the original image pair can be used to train convolutional block 506 for demosaicing ML model 520. For example, the original image can be used as a baseline truth, and the corresponding Bayer array image can be input into the demosaicing ML model 520 during training. During training, depthwise convolutional block 504 and depth-to-space block 508 can operate in the same way as during inference. Convolutional block 506 may attempt to perform interpolation operations, and the output can be compared with the original image to compute a loss function used to adjust the weights of convolutional block 506. The weights of depthwise convolutional block 504 and depth-to-space block 508 are not adjusted.

[0071] In some cases, the operations performed by convolution block 506 can be performed on the MAC unit of the NPU. As discussed above, the MAC unit of the NPU can be optimized to accelerate convolution-based operations, and when performed on the MAC unit of the NPU, the operations of convolution block 506 can take advantage of such optimizations.

[0072] In some cases, the output color channels of convolutional block 506 can be input into depth-to-space block 508. Depth-to-space block 508 rearranges the data from the depth dimension into blocks of spatial dimension data. For example, color channels can be treated as depth dimensions, and depth-to-space block 508 can perform a series of index operations (e.g., adjusting memory offset values) to obtain a processed image (e.g., a demosaic image, an RGB color image, a full-color image, etc.) for output 510. For example, as indicated above, convolutional block 506 can output multiple output channels for each input channel (e.g., the number of color channels based on the pixel pattern of the input image 502), and depth-to-space block 508 can arrange (e.g., merge, rearrange) multiple output channels to obtain the same number of channels as the input channels. As a more specific example, convolutional block 506 can output 12 channels for three channel inputs (e.g., the color channels based on the Bayer array input image), and depth-to-space block 508 can rearrange the 12 channels back into 3 channels. In some cases, this can be done by adjusting the offset used to locate the next value for a color channel (e.g., after reading x red channel values, the next red channel value could be located at y). In some cases, an NPU can be used to recalculate these offsets. In some cases, the depth-to-space block 508 may be untrainable.

[0073] In some cases, the convolutional block 506 and the depth-to-space block 508 pair can be replaced by a convolutional block 506 and a transposed convolutional block pair (not shown). The convolutional block 506 and the transposed convolutional block pair de-mosaic and upsample the stitched color channels output from the depth convolutional block 504. The transposed convolutional block may include one or more layers performing the transposed convolution operation. The transposed convolution operation is essentially a dimension-inverted convolution operation. In some cases, the number of output channels of the convolutional block 506 can also be adjusted. In some cases, the transposed convolution operation can be trainable.

[0074] In some cases, convolutional block 506 can be replaced by a bilinear resizing operation, and depth-to-space block 508 can be replaced by a convolutional block (not shown). The bilinear resizing operation can interpolate the output from depth convolutional block 504 and can be performed iteratively (e.g., without using ML). The convolutional block can be trainable and the output of the bilinear resizing operation can be further filtered to perform a mosaic operation.

[0075] Figure 6 This is a flowchart illustrating a process 600 for demosaicing image data according to various aspects of the present disclosure. Process 600 may be performed by a computing device (or apparatus) or components of a computing device (e.g., chipset, codec, etc.) (such as...). Figure 1The image capture and processing system 100 performs the operation. The computing device can be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an extended reality (AR) device), a vehicle or a component or system of a vehicle, a drone, a flying object, or other type of computing device. The operation of process 600 can be implemented as a process in one or more processors (e.g., [processor name missing]). Figure 1 Image processor 150 Figure 1 The host processor 152, Figure 1 NPU 158, Figure 8 Software components that execute and run on the processor 810, any combination thereof, and / or other processors. In some cases, the operation of process 600 may be performed by a SoC (such as...) Figure 7 Systems with SoC 705 and / or having Figure 8 The computing system is implemented using the architecture of the 800 system.

[0076] At box 602, a computing device (or a component thereof) may perform a depth convolution operation on image data and a depth convolution filter having predetermined parameter values ​​to obtain multiple color channels of the image data. For example, a depth convolution block of an ML model (e.g., Figure 5 A depth convolution block 504 can perform depth convolution on image data. In some cases, the image data is raw image data. For example, raw image data may be unprocessed data obtained from an image sensor. In some cases, the raw image data includes a Bayer array (e.g., R, G, G, B). In some cases, the depth convolution operation is performed using at least one depth convolution filter that includes predetermined parameter values. For example, the predetermined parameter values ​​may be predetermined during research and development and / or manufacturing processes. In some cases, the predetermined parameter values ​​of the at least one depth convolution filter used for the depth convolution operation are predetermined based on the color filter of the image sensor used to capture the image data. In some cases, the predetermined parameter values ​​of the at least one depth convolution filter are set to 0 or 1. Using 1 and 0 as the only values ​​in the filter can help to efficiently extract color values ​​from the image data while reducing the power consumption of the MAC unit. In some examples, multiple sets of predetermined parameter values ​​may be predetermined for the image sensor. In some cases, predetermined parameter values ​​of at least one depth convolution filter are determined to extract pixel color values ​​from the image data. In some cases, the color filters are based on Bayer arrays, and at least one of the deep convolutional filters includes a 2x2 kernel size and a stride of 2. In other cases, the deep convolutional operation is performed by a machine learning model for image demosaicing.

[0077] At box 604, a computing device (or a component thereof) may perform convolution operations on multiple color channels to obtain processed color channels. For example, an ML model may include one or more convolutional blocks (e.g., Figure 5 The convolutional block 506 is used to perform convolution operations. In some cases, the convolution operation is performed using multiple convolutional filters, the parameters of which are tuned based on the training process. In some cases, the number of multiple convolutional filters is determined at least based on a magnification factor. For example, a magnification factor can be used to magnify stitched color channels to the original resolution of the image data. In some cases, depthwise convolution operations and convolution operations are performed on a multiply-accumulate hardware unit. In some cases, the multiply-accumulate hardware unit is included in a neural processing unit, a graphics processing unit, and / or a central processing unit. In some cases, the machine learning model is configured to perform depthwise convolution operations, convolution operations, and arrange the processed multiple color channels into a demosaic image on a GPU or CPU. In some examples, the neural processing unit includes an application-specific integrated circuit (ASIC), and the ASIC is incorporated into a machine learning model configured to perform depthwise convolution operations, convolution operations, and arrange the processed multiple color channels into a demosaic image. In some cases, the NPU may be integrated into the image sensor.

[0078] At box 606, a computing device (or a component thereof) can arrange multiple processed color channels into a demosaic image. For example, one or more layers of an ML model can arrange (e.g., merge, rearrange) multiple output channels to obtain the same number of channels as the input channels. In some cases, this is achieved by performing depth-to-space operations (e.g., via...). Figure 5 The depth-to-space block (508) arranges the processed multiple color channels into RGB channels. In some examples, the depth-to-space operation involves arranging blocks of data containing the processed multiple color channels in space to generate a demosaic image. In some cases, the processed multiple color channels are upsampled by performing a transposed convolution operation.

[0079] At box 608, the computing device (or a component thereof) may output a de-mosaic image.

[0080] Figure 7An example implementation of a System-on-a-Chip (SOC) 705 is illustrated, which may include a Central Processing Unit (CPU) 710 or a multi-core CPU configured to perform one or more of the functions described herein. In some cases, the SOC 705 may be based on the ARM instruction set. In some cases, the CPU 710 may be similar to a processor 764. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., weighted neural networks), latency, frequency bin information, task information, and other information may be stored in a memory block associated with a Neural Processing Unit (NPU) 725, a memory block associated with the CPU 710, a memory block associated with a Graphics Processing Unit (GPU) 715, a memory block associated with a Digital Signal Processor (DSP) 706, a memory block 785, and / or may be distributed across multiple blocks. Instructions executed at the CPU 710 may be loaded from the program memory associated with the CPU 710 or may be loaded from memory block 785.

[0081] The SOC 705 may also include additional processing blocks tailored for specific functions, such as a GPU 715, a DSP 706, a connectivity block 735 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, and Bluetooth connectivity, etc.), and a multimedia processor 745 capable of, for example, detecting and recognizing gestures. In one implementation, the NPU is implemented within the CPU 710, DSP 706, and / or GPU 715. The SOC 705 may also include a sensor processor 755, an image signal processor (ISP) 775, and / or a navigation module 795, which may include a global positioning system. In some cases, the navigation module 795 may resemble the navigation component 756, and the sensor processor 755 may accept input from, for example, one or more sensors 758. In some cases, the connectivity block 735 may resemble the radio module 772.

[0082] Figure 8 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Specifically, Figure 8 An example of a computing system 800 is illustrated. This computing system can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 805. The connection 805 can be a physical connection using a bus, or a direct connection to a processor 810, such as in a chipset architecture. The connection 805 can also be a virtual connection, a networking connection, or a logical connection.

[0083] In some examples, the computing system 800 is a distributed system, wherein the functionality described herein may be distributed across a single data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a plurality of such components, each performing some or all of the functionality of the described components. In some cases, these components may be physical or virtual devices.

[0084] Example computing system 800 includes at least one processing unit (CPU or processor) 810 and a connection 805 that couples various system components, including system memory 815 such as read-only memory (ROM) 820 and random access memory (RAM) 825, to processor 810. Computing system 800 may include a cache 812 of high-speed memory that is directly connected to, closely proximate to, or integrated into processor 810.

[0085] Processor 810 may include any general-purpose processor and hardware or software services (such as services 832, 834, and 836 stored in storage device 830 and configured to control processor 810), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 810 may be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0086] To enable user interaction, the computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, a camera, an accelerometer, a gyroscope, etc. The computing system 800 may also include an output device 835 that can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows a user to provide multiple types of input / output to communicate with the computing system 800. The computing system 800 may include a communication interface 840 that typically governs and manages user input and system output. The communication interface can perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including wired and / or wireless transceivers utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ® Lightning ® Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low-power (BLE) wireless signal transmission, IBEACON ®Wireless signal transmission, including radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.10 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 840 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 800 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware configuration, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware configurations as they are developed.

[0087] Storage device 830 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital multifunction disks, cartridges, floppy disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM discs, rewritable CD discs, DVD discs, Blu-ray discs (BDD discs), holographic discs, another optical medium, secure digital (SD) cards, microSD cards, Memory Sticks. ®Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0088] Storage device 830 may include software services, servers, etc., which enable the system to perform functions when the code defining such software is executed by processor 810. In some examples, hardware services that perform specific functions may include software components for performing functions stored in computer-readable media connected to necessary hardware components such as processor 810, connection 805, output device 835, etc.

[0089] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact optical discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0090] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0091] Specific details are provided in the foregoing description to provide a thorough understanding of the examples provided herein. However, those skilled in the art will understand that the examples can be implemented without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the examples with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the examples.

[0092] The examples above may be described as processes or methods, depicted as flowcharts, diagrams, data flow diagrams, structure diagrams, or block diagrams. While a flowchart may describe operations as a sequential process, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to its calling function or the main function.

[0093] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0094] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor performs the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0095] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0096] In the foregoing description, aspects of this application have been described with reference to specific examples of this application, but those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts can be implemented and employed in a variety of other ways, and the appended claims are intended to be construed as including such variations unless limited by prior art. Various features and aspects of the applications described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative examples, the methods may be performed in a different order than that described.

[0097] Those skilled in the art will understand that, without departing from the scope of this description, the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("<"), respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0098] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0099] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0100] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0101] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0102] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0103] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0104] Claims using phrases such as "at least one processor, the at least one processor being configured to," "at least one processor being configured to," "one or more processors, the one or more processors being configured to," or "one or more processors being configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0105] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0106] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0107] The exemplary aspects of this disclosure include: Aspect 1. An apparatus for demosaicing one or more images, the apparatus comprising: at least one memory configured to store image data; and at least one processor coupled to the at least one memory and configured to: perform a depthwise convolution operation on the image data and a depthwise convolution filter having predetermined parameter values ​​to obtain a plurality of color channels of the image data; perform a convolution operation on the plurality of color channels to obtain a processed plurality of color channels; arrange the processed plurality of color channels into a demosaic image; and output the demosaic image.

[0108] Aspect 2. The apparatus according to aspect 1, wherein the image data is raw image data.

[0109] Aspect 3. The apparatus according to aspect 2, wherein the raw image data includes a Bayer array.

[0110] Aspect 4. The apparatus according to any one of Aspects 1 to 3, wherein the at least one processor is configured to perform the depth convolution operation using at least one depth convolution filter including predetermined parameter values.

[0111] Aspect 5. The apparatus according to aspect 4, wherein the predetermined parameter values ​​of the at least one deep convolution filter for the deep convolution operation are predetermined based on a color filter of an image sensor used to capture the image data.

[0112] Aspect 6. The apparatus according to aspect 5, wherein the color filter is based on a Bayer array, and wherein the at least one deep convolutional filter comprises a kernel size of 2x2 and a stride of 2.

[0113] Aspect 7. The apparatus according to any one of Aspects 4 to 6, wherein the predetermined parameter values ​​of the at least one depth convolution filter are determined to extract pixel color values ​​from the image data.

[0114] Aspect 8. The apparatus according to aspect 7, wherein the predetermined parameter value of the at least one deep convolution filter is set to 0 or 1.

[0115] Aspect 9. The apparatus according to any one of Aspects 1 to 8, wherein the at least one processor is configured to perform the convolution operation using a plurality of convolutional filters, wherein the parameters of the plurality of convolutional filters are tuned based on a training process.

[0116] Aspect 10. The apparatus according to aspect 9, wherein the number of the plurality of convolutional filters is determined at least based on the amplification factor.

[0117] Aspect 11. The apparatus according to any one of Aspects 1 to 10, wherein the at least one processor is configured to arrange the processed plurality of color channels into RGB channels based on depth-to-space operations.

[0118] Aspect 12. The apparatus according to aspect 11, wherein, in order to perform the depth-to-space operation, the at least one processor is configured to spatially arrange data blocks of the processed plurality of color channels to generate the demosaic image.

[0119] Aspect 13. The apparatus according to any one of Aspects 1 to 12, wherein the at least one processor is configured to upsample the processed plurality of color channels based on a transposed convolution operation.

[0120] Aspect 14. The apparatus according to any one of Aspects 1 to 13, wherein the depth convolution operation is performed by a machine learning model for image demosaicing.

[0121] Aspect 15. The apparatus according to any one of Aspects 1 to 14, wherein the at least one processor is further configured to perform the depthwise convolution operation and the convolution operation on a multiply-accumulate hardware unit.

[0122] Aspect 16. The apparatus according to aspect 15, wherein the at least one processor includes a neural processing unit, and wherein the neural processing unit includes the multiplication and accumulation hardware unit.

[0123] Aspect 17. The apparatus of aspect 16, wherein the neural processing unit includes an application-specific integrated circuit (ASIC), and wherein the ASIC is incorporated into a machine learning model configured to perform the depthwise convolution operation, the convolution operation, and to arrange the processed plurality of color channels into the demosaic image.

[0124] Aspect 18. The apparatus according to aspect 15, wherein the at least one processor includes a graphics processing unit (GPU) or a central processing unit (CPU), the GPU or CPU including the multiply and accumulate hardware unit, and wherein the machine learning model configured to perform the depthwise convolution operation, the convolution operation and to arrange the processed plurality of color channels into the demosaic image is executed on the GPU or CPU.

[0125] Aspect 19. The apparatus according to any one of aspects 1 to 18, wherein the apparatus comprises one or more image sensors.

[0126] Aspect 20. The apparatus according to aspect 19, wherein the at least one processor is integrated into the one or more image sensors.

[0127] Aspect 21. The apparatus according to any one of aspects 19 to 20, wherein the one or more image sensors include one or more cameras.

[0128] Aspect 22. A method for demosaicing an image, the method comprising: performing a depthwise convolution operation on image data and a depthwise convolution filter having predetermined parameter values ​​to obtain a plurality of color channels of the image data; performing a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arranging the plurality of processed color channels into a demosaic image; and outputting the demosaic image.

[0129] Aspect 23. The method according to aspect 22, wherein the image data is raw image data.

[0130] Aspect 24. The method according to aspect 23, wherein the raw image data includes a Bayer array.

[0131] Aspect 25. The method according to any one of Aspects 22 to 24, wherein the depth convolution operation is performed using at least one depth convolution filter comprising predetermined parameter values.

[0132] Aspect 26. The method according to aspect 25, wherein the predetermined parameter values ​​of the at least one deep convolution filter used for the deep convolution operation are predetermined based on the color filter of the image sensor used to capture the image data.

[0133] Aspect 27. The method according to aspect 26, wherein the color filter is based on a Bayer array, and wherein the at least one deep convolutional filter comprises a kernel size of 2x2 and a stride of 2.

[0134] Aspect 28. The method according to any one of Aspects 25 to 27, wherein the predetermined parameter values ​​of the at least one depth convolution filter are determined to extract pixel color values ​​from the image data.

[0135] Aspect 29. The method according to aspect 28, wherein the predetermined parameter value of the at least one depthwise convolutional filter is set to 0 or 1.

[0136] Aspect 30. The method according to any one of Aspects 22 to 29, wherein the convolution operation is performed using a plurality of convolution filters, wherein the parameters of the plurality of convolution filters are tuned based on a training process.

[0137] Aspect 31. The method according to aspect 30, wherein the number of the plurality of convolutional filters is determined at least based on the amplification factor.

[0138] Aspect 32. The method according to any one of aspects 22 to 31, wherein the processed plurality of color channels are arranged into RGB channels by performing a depth-to-space operation.

[0139] Aspect 33. The method according to aspect 32, wherein the depth-to-space operation includes spatially arranging data blocks of the processed multiple color channels to generate the demosaic image.

[0140] Aspect 34. The method according to any one of Aspects 22 to 33, wherein the processed plurality of color channels are upsampled by performing a transposed convolution operation.

[0141] Aspect 35. The method according to any one of Aspects 22 to 34, the method further comprising performing the depthwise convolution operation and the convolution operation on a multiplication and accumulation hardware unit.

[0142] Aspect 36. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by at least one processor, causing the at least one processor to: perform a depth convolution operation on image data and a depth convolution filter having predetermined parameter values ​​to obtain a plurality of color channels of the image data; perform a convolution operation on the plurality of color channels to obtain a plurality of processed color channels; arrange the plurality of processed color channels into a demosaic image; and output the demosaic image.

[0143] Aspect 37. The non-transitory computer-readable medium of claim 36, wherein the instructions cause the at least one processor to perform one or more operations according to any one of aspects 22 to 35.

[0144] Aspect 38: An apparatus for demosaicing an image, the apparatus comprising components for performing one or more operations according to any one of aspects 22 to 35.

Claims

1. An apparatus for demosaicing one or more images, the apparatus comprising: At least one memory, the at least one memory being configured to store image data; and At least one processor, the at least one processor being coupled to the at least one memory and configured to: A depth convolution operation is performed on the image data and a depth convolution filter with predetermined parameter values ​​to obtain multiple color channels of the image data; Perform convolution operations on the multiple color channels to obtain the processed multiple color channels; Arrange the processed multiple color channels into a de-mosaic image; as well as Output the depixelated image.

2. The apparatus according to claim 1, wherein the image data is raw image data.

3. The apparatus of claim 2, wherein the raw image data comprises a Bayer array.

4. The apparatus of claim 1, wherein the at least one processor is configured to perform the depth convolution operation using at least one depth convolution filter including predetermined parameter values.

5. The apparatus of claim 4, wherein the predetermined parameter values ​​of the at least one deep convolution filter for the deep convolution operation are predetermined based on a color filter of an image sensor used to capture the image data.

6. The apparatus of claim 5, wherein the color filter is based on a Bayer array, and wherein the at least one deep convolutional filter comprises a kernel size of 2x2 and a stride of 2.

7. The apparatus of claim 4, wherein the predetermined parameter values ​​of the at least one depth convolution filter are determined to extract pixel color values ​​from the image data.

8. The apparatus of claim 7, wherein the predetermined parameter value of the at least one deep convolutional filter is set to 0 or 1.

9. The apparatus of claim 1, wherein the at least one processor is configured to perform the convolution operation using a plurality of convolutional filters, wherein the parameters of the plurality of convolutional filters are tuned based on a training process.

10. The apparatus of claim 9, wherein the number of the plurality of convolutional filters is determined at least based on the amplification factor.

11. The apparatus of claim 1, wherein the at least one processor is configured to arrange the processed plurality of color channels into RGB channels based on depth-to-space operations.

12. The apparatus according to claim 11, wherein, In order to perform the depth-to-space operation, the at least one processor is configured to spatially arrange data blocks of the processed multiple color channels to generate the demosaic image.

13. The apparatus of claim 1, wherein the at least one processor is configured to upsample the processed plurality of color channels based on a transposed convolution operation.

14. The apparatus of claim 1, wherein the at least one processor is configured to perform the depthwise convolution operation using a machine learning model configured to perform image demosaicing.

15. The apparatus of claim 1, wherein the at least one processor is further configured to perform the depthwise convolution operation and the convolution operation on a multiply-accumulate hardware unit.

16. The apparatus of claim 15, wherein the at least one processor includes a neural processing unit, and wherein the neural processing unit includes the multiplication and accumulation hardware unit.

17. The apparatus of claim 16, wherein the neural processing unit comprises an application-specific integrated circuit (ASIC), and wherein the ASIC is incorporated into a machine learning model configured to perform the depthwise convolution operation, the convolution operation, and to arrange the processed plurality of color channels into the demosaic image.

18. The apparatus of claim 15, wherein the at least one processor comprises a graphics processing unit (GPU) or a central processing unit (CPU), the GPU or CPU comprising the multiply and accumulate hardware unit, and wherein the machine learning model configured to perform the depthwise convolution operation, the convolution operation, and to arrange the processed plurality of color channels into the demosaic image is executed on the GPU or CPU.

19. The apparatus of claim 1, wherein the apparatus comprises one or more image sensors.

20. The apparatus of claim 19, wherein the at least one processor is integrated into the one or more image sensors.

21. The apparatus of claim 19, wherein the one or more image sensors comprise one or more cameras.

22. A method for demosaicing an image, the method comprising: A depth convolution operation is performed on image data and a depth convolution filter with predetermined parameter values ​​to obtain multiple color channels of the image data; Perform convolution operations on the multiple color channels to obtain the processed multiple color channels; Arrange the processed multiple color channels into a de-mosaic image; as well as Output the depixelated image.

23. The method of claim 22, wherein the image data is raw image data.

24. The method of claim 22, wherein the depthwise convolution operation is performed using at least one depthwise convolution filter comprising predetermined parameter values.

25. The method of claim 24, wherein the predetermined parameter values ​​of the at least one deep convolution filter used for the deep convolution operation are predetermined based on a color filter of an image sensor used to capture the image data.

26. The method of claim 24, wherein the predetermined parameter values ​​of the at least one depth convolution filter are determined to extract pixel color values ​​from the image data.

27. The method of claim 22, wherein the convolution operation is performed using a plurality of convolutional filters, wherein the parameters of the plurality of convolutional filters are tuned based on the training process.

28. The method of claim 22, wherein the processed plurality of color channels are arranged into RGB channels by performing a depth-to-space operation.

29. The method of claim 22, further comprising performing the depthwise convolution operation and the convolution operation on a multiplication and accumulation hardware unit.

30. A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to: A depth convolution operation is performed on image data and a depth convolution filter with predetermined parameter values ​​to obtain multiple color channels of the image data; Perform convolution operations on the multiple color channels to obtain the processed multiple color channels; as well as Arrange the processed multiple color channels into a de-mosaic image; as well as Output the depixelated image.

Images