Image processor formed in memory cell array

Abstract

Apparatuses, systems, and methods related to an image processor formed in an array of memory cells are described. An image processor as described herein is configured to reduce complexity and power consumption and / or increase data access bandwidth by performing image processing in the array of memory cells relative to image processing by a host processor external to the memory array. For example, one apparatus described herein includes a sensor circuit configured to provide an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel; and an image processor formed in the memory cell array. The image processor is coupled to the sensor circuit to receive the plurality of bits of the input vector. The image processor is configured to determine an output vector for color correction by performing a matrix multiplication on the input vector and a parameter matrix to perform a color correction operation in the array.

Description

[0001] Relevant information of divisional application

[0002] This application is a divisional application of a Chinese patent application with application number 201980037979.0, application date April 9, 2019, and invention title "Image Processor Formed in a Memory Cell Array". Technical Field

[0003] The present disclosure generally relates to semiconductor memories and methods, and more particularly, to devices, systems, and methods for an image processor formed in a memory cell array. Background Art

[0004] Memory resources are typically provided as internal semiconductor integrated circuits in a computer or other electronic system. There are many different types of memories, including volatile and non-volatile memories. Volatile memory may require power to maintain its data (e.g., host data, error data, etc.). Volatile memory may include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), and thyristor random access memory (TRAM), among others. Non-volatile memory may provide persistent data by storing the stored data when not powered. Non-volatile memory may include NAND flash memory, NOR flash memory, and resistive variable memories such as phase change random access memory (PCRAM) and resistive random access memory (ReRAM), ferroelectric random access memory (FeRAM), and magnetoresistive random access memory (MRAM), such as spin torque transfer random access memory (STT RAM), among others. Summary of the Invention

[0005] In one aspect, the present disclosure relates to an image processing device including: a sensor circuit configured to provide an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel; an image processor formed in a memory cell array; and wherein the image processor: is coupled to the sensor circuit to receive the plurality of bits of the input vector; and is configured to perform a color correction operation in the array by performing matrix multiplication on the input vector and a parameter matrix to determine an output vector for color correction.

[0006] In another aspect, the present disclosure relates to an image processing system, comprising: an optical sensor arranged in a color filter array for digitally acquiring a color image, wherein the optical sensor is configured to provide a plurality of bits corresponding to a plurality of color components of an image pixel; an image processor formed in an array of memory cells as a plurality of serially coupled matrix multiplication units (MMUs); and wherein the image processor is configured to: be coupled to the color filter array to receive the plurality of bits; perform one of a plurality of color correction operation sequences on each of the corresponding plurality of serially coupled MMUs; and perform matrix multiplication on an input vector and a parameter matrix on each of the corresponding plurality of serially coupled MMUs to determine an output vector for color correction.

[0007] In another aspect, the present disclosure relates to an image processing system, comprising: a digital image sensor for digitally acquiring a color image, wherein each of the sensors is configured to provide a data value indicative of one of a plurality of color components of an image pixel; an image processor formed in association with an array of memory cells to include: a single matrix multiplication unit (MMU); parameter matrix components configured to store a plurality of alternative parameter matrices; a control unit configured to select from the plurality of alternative parameter matrices and direct an input to the single MMU of the selected parameter matrix; and wherein the image processor is configured to: be coupled to the plurality of sensors to receive data values indicative of each level of the plurality of color components; perform a sequence of a plurality of color correction operations on the single MMU; and perform matrix multiplication on an input vector and the selected parameter matrix input to the single MMU by the control unit from the parameter matrix components to determine an output vector for color correction.

[0008] In another aspect, the present disclosure relates to a method of operating a memory device for image processing, comprising: transmitting an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel to an image processor formed in an array of memory cells; performing matrix multiplication on the input vector and a parameter matrix by the image processor formed in the array; and determining, by the image processor formed in the array, an output vector including a plurality of color-corrected components based at least in part on the matrix multiplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a schematic diagram illustrating an example of a resource implementation that can be used to include an image processor formed in an array of memory cells according to various embodiments of the present disclosure.

[0010] Figure 2A block diagram of an example of a portion of a resistive random access memory (ReRAM) array that can operate as an image processor according to multiple embodiments of the present disclosure.

[0011] Figure 3 An example of a parameter matrix that can be used by an image processor formed in a memory cell array according to multiple embodiments of the present disclosure.

[0012] Figure 4 A block diagram of an example of multiple color correction operations that can be performed by multiple matrix multiplication units (MMUs) of an image processor formed in a memory cell array according to multiple embodiments of the present disclosure.

[0013] Figure 5 A block diagram of an example of multiple color correction operations that can be performed by corresponding multiple MMUs of an image processor formed in a memory cell array according to multiple embodiments of the present disclosure.

[0014] Figure 6 A block diagram of an example of a configuration for performing multiple color correction operations on a single MMU of an image processor formed in a memory cell array according to multiple embodiments of the present disclosure.

[0015] Figure 7 A flowchart of an example of an image processor formed in a memory cell array that performs color correction according to multiple embodiments of the present disclosure. Detailed Description

[0016] The present disclosure includes systems, devices, and methods associated with an image processor formed in a memory cell array. In multiple embodiments, the device includes sensor circuitry configured to provide an input vector as multiple bits corresponding to multiple color components of an image pixel; and an image processor formed in the memory cell array. The image processor is coupled to the sensor circuitry to receive the multiple bits of the input vector. The image processor is configured to perform a color correction operation in the memory cell array by performing matrix multiplication on the input vector and a parameter matrix to determine an output vector for color correction.

[0017] As the demand for higher image resolution increases, the data access bandwidth from a memory (e.g., DRAM) to a processing unit (e.g., on a host that can be indirectly coupled to the memory) of an image signal processor (ISP) gradually increases. This increase in data access bandwidth can lead to excessive power consumption of such an ISP (e.g., for mobile and / or remote devices such as smart phones).

[0018] The present disclosure describes various embodiments of an image processor to perform color correction processing in a memory cell array to overcome such potential difficulties and to provide other advantages. Along with the absence of data feedback loops and / or associated data dependencies between color correction operations, various embodiments of the signal processing implementation for color correction described herein may allow most (e.g., all) of such processing (e.g., its computing units and / or control units) to be moved to a memory chip (e.g., associated with and / or within a RAM array) in order to reduce the complexity of memory access.

[0019] In the architecture described herein, the image processor may be formed as part of a RAM (ReRAM) to act as a processor-in-memory (PIM). In various embodiments, this image processor may be referred to as an in-memory color correction processor. Relative to other ISP implementations, this image processor may reduce the power consumption for color correction without causing a performance degradation, without adding significantly more components to the memory chip. For example, the crossbar switch architecture of ReRAM memory cells can not only store (e.g., for reading and / or writing) data, but can also be used to perform arithmetic operations such as addition, subtraction, multiplication, and / or division on operands via multiple operations, in addition to matrix multiplication described herein, without incorporating additional components.

[0020] The present disclosure's Figure 2 illustrates ReRAM cell and crossbar switch configurations for computing the color correction operations described herein. For example, a crossbar switch with a 3×3 ReRAM cell configuration can be used to perform matrix multiplication on a 3×3 matrix (parameter matrix) with a 3×1 vector and then provide the output as a 3×1 vector. As described herein, without modifying the inherent configuration of the ReRAM crossbar switch array, a crossbar switch of 3×3 = 9 ReRAM cells can be used to facilitate or replace the MMU of an ISP.

[0021] According to the pipelined image processor configuration described in conjunction with Figure 5 multiple crossbar switch arrays (e.g., multiple 3×3 MMUs) can be configured to perform multiple ISP stages of color correction for a corresponding number of multiple pixels simultaneously (e.g., during the same clock cycle). In various embodiments (e.g., as described in conjunction with Figure 4 and 5In the case shown and described, the number of multiple MMUs, multiple ISP stages, and multiple pixels can each be 7. For example, the raw data of the first pixel (e.g., a 3×1 vector corresponding to RGB color components) can be input to the first crossbar switch for processing in the first stage. A 3×1 vector can be output from the first crossbar switch and input to the second crossbar switch to perform the second stage of processing on the first pixel. At the same time, the first crossbar switch can perform the first stage of processing on the second pixel. Continuing this pipelined mode, all seven processing stages for seven different pixels can be performed substantially simultaneously.

[0022] Corresponding to the image processor configuration described in conjunction with Figure 6 For the described image processor configuration, one crossbar switch can be used to perform seven processing stages on each pixel. For example, the first stage (e.g., a clock cycle) can be to input the raw data of one pixel (e.g., an RGB vector) to one crossbar switch for processing in the first stage. The processing in the second stage can be to input the output from the first stage (e.g., a 3×1 vector) back into the same crossbar switch to perform the second stage of processing on the same pixel. After seven stages, all seven processing stages for the same pixel can be completed on a crossbar switch array.

[0023] Therefore, due to the inherent signal processing capabilities of ReRAM, the color correction processing of the ISP can be performed by the memory and / or on the memory. In addition, for example, by using a crossbar ReRAM architecture, additional processing components (e.g., unnecessary) for color correction can be reduced because the crossbar of ReRAM can not only store data but also be used to perform matrix multiplication and other arithmetic operations described herein. Therefore, the image processors described herein can facilitate reducing power consumption and / or increasing the data access bandwidth related to image signals and / or color correction processing.

[0024] The figures in this document follow a numbering convention, where the first one or more digits of the reference numeral correspond to the figure number, and the remaining digits identify the element or component in the figure. Similar elements or components between different figures can be identified by using similar numbers. For example, 104 can refer to Figure 1 the element "04" in Figure 5 and a similar element can be labeled 504 in

[0025] Figure 1 is a schematic diagram of an example of a resource implementation that can be used to include an image processor formed in a memory cell array according to multiple embodiments of the present disclosure. Figure 1 The resource implementation 100 illustrated in

[0026] Resource implementation 100 described herein includes sensor circuitry 102 configured to provide an input vector as a plurality of bits 103 corresponding to a plurality of color components of an image pixel. In various embodiments, the sensor circuitry 102 can be or can include a digital image sensor and / or a light sensor arranged as a color filter array (e.g., a Bayer filter and other possible configurations) for digital acquisition of a color image. The digital image sensor and / or the light sensor can be configured to provide a plurality of bits 103 corresponding to a plurality of color components of an image pixel. For example, each of the sensors can be configured to provide a data value indicative of a level of one of the plurality of color components of an image pixel.

[0027] Resource implementation 100 described herein also includes an image processor 104 formed in a memory cell array (e.g., as shown and described in connection with Figure 2 ). The image processor 104 can be coupled to (e.g., directly via a bus and without intermediate processing circuitry) the sensor circuitry 102 to receive the plurality of bits 103 of the input vector, the plurality of bits corresponding to a plurality of color components of an image pixel sensed and / or acquired by the sensor circuitry 102. The image processor 104 can be configured to perform a color correction operation by performing matrix multiplication on the input vector (e.g., as shown at 331 and described in connection with Figure 3 and a parameter matrix (e.g., as shown at 333 and described in connection with Figure 3 to determine an output vector for color correction (e.g., as shown at 334 and described in connection with Figure 3 and elsewhere herein).

[0028] The memory cell array including the image processor 104 can be formed to include at least a portion of a RAM array 105. The image processor 104 can be formed in the RAM array 105 to include a MMU 106. The MMU 106 can be formed in the array as a plurality of memory cells (e.g., as shown and described in connection with Figure 2 ), the plurality of memory cells formed as the MMU 106. As described herein, in various embodiments, the number of memory cells can correspond to the number of parameter entries in the parameter matrix. The memory cells included in the image processor 104 and / or the MMU 106 can be configured to perform (e.g., via matrix multiplication) the determination of the color-corrected output vector. In various embodiments, the determination of the color-corrected output vector can be performed in memory (e.g., in the array) without the memory cells being coupled to a sensing circuit including sense amplifiers (not shown).

[0029] The number of memory cells of the MMU 106 can be formed to correspond to the parameter matrix (e.g., as shown at 333 and described in connection with Figure 3the number of parameter entries in the (described). In multiple embodiments, multiple color components of an image pixel can correspond to three color components. The three color components can be, for example, red (R), green (G), and blue (B), as well as other possible numbers and / or colors of the color components of an image pixel (such as cyan (C), magenta (M), and yellow (Y), etc.). The input vector (e.g., as shown and described in conjunction with Figure 3 the described) multiple bits 103 can be three bits to correspond to three color components. The image processor 104 can be configured to perform matrix multiplication on the input vector and the parameter matrix to generate an output vector (e.g., as shown and described in conjunction with Figure 3 the described), to include three color-corrected bits in multiple embodiments. In multiple embodiments, a fast Fourier transform (FFT) can be performed for the matrix multiplication.

[0030] The resource implementation 100 described herein can also include a controller 107, which is configured to direct (e.g., directly) the input of the input vector from the sensor circuit 102 to the image processor 104 and / or the MMU 106. The controller 107 can also be configured to direct the execution of multiple color correction operations in the array (e.g., as described in conjunction with Figure 4 the described) through the image processor 104 to determine the color-corrected output vector.

[0031] The resource implementation 100 described herein can also include a display processor 108. The display processor 108 can be configured to perform other ISP operations after the image processor 104 performs color correction operations to generate the output vector as described herein. In multiple embodiments, the output from the display processor 108 can be an image and / or a series of images that can be displayed by the screen 109 (e.g., user interface).

[0032] In some embodiments, the image processor 104 and / or the MMU 106 can be implemented on multiple memory resources. As used herein, "memory resources" is a general term intended to include at least memory (e.g., memory cells) such as memory arranged in multiple bank groups, banks, bank segments, sub-arrays, and / or rows of multiple memory devices. In multiple embodiments, the memory resources can be or can include multiple volatile memory devices formed as and / or operable as RAM, DRAM, SRAM, SDRAM, and / or TRAM, as well as other types of volatile memory devices. Alternatively or additionally, in multiple embodiments, the memory resources can be or can include multiple non-volatile memory devices formed as and / or operable as NAND, NOR, PCRAM, ReRAM, FeRAM, MRAM, and / or STT RAM, as well as other types of non-volatile memory devices.

[0033] Figure 2 is a block diagram of an example of a portion of a ReRAM array that can operate as an image processor 104 and / or an MMU 106 in accordance with multiple embodiments of the present disclosure. In multiple embodiments, an image processor (e.g., as shown at 104, 504-1, and 604-2 and described in connection with Figure 1 , 5 and 6, respectively) can be formed in a memory cell array (e.g., as shown at 105 and described in connection with Figure 1 the RAM array described) and / or formed as the memory cell array. Figure 2 The portion of the ReRAM array illustrated in Figure 2 can correspond to an embodiment of a single MMU 206, but embodiments of the image processor 104 described herein are not limited to a single MMU, an MMU formed as a ReRAM array, and / or Figure 2 the multiple ReRAM memory cells shown in

[0034] As Figure 2 shown in Figure 2 , in multiple embodiments, the memory cell array of a single MMU 206 can be formed as a 3×3 array of ReRAM cells in a crossbar switch configuration 211. Thus, each ReRAM memory cell 217 can be coupled to a first wire 213 and a second wire 215 and / or formed between the first wire 213 and the second wire 215. In multiple embodiments, each of the first wires 213 can correspond to a word line, and each of the second wires 215 can correspond to a bit line, or vice versa. For example, in the embodiment of the 3×3 crossbar switch array 211 shown in Figure 2 , the MMU 206 includes ReRAM cells 217-1-1, 217-1-2, and 217-1-3 coupled to wire 213-1, ReRAM cells 217-2-1, 217-2-2, and 217-2-3 coupled to wire 213-2, and ReRAM cells 217-3-1, 217-3-2, and 217-3-3 coupled to wire 213-3. Due to the configuration of the 3×3 crossbar switch array 211, the ReRAM cells 217-1-1, 217-2-1, and 217-3-1 are coupled to wire 215-1, the ReRAM cells 217-1-2, 217-2-2, and 217-3-2 are coupled to wire 213-2, and the ReRAM cells 217-1-3, 217-2-3, and 217-3-3 are coupled to wire 215-3.

[0035] An example memory cell included in a crossbar array 211 configuration of ReRAM cells for a single MMU 206 is shown in more detail at 217-2-3 in the array and next to the array. Details of the ReRAM cell 217-2-3 are shown in multiple embodiments, and the ReRAM cell can have an architecture that includes a top electrode 220 (e.g., which can be the same as or different from wire 213-2), the top electrode being coupled to a source component (not shown) to apply a specific voltage potential 219. The ReRAM cell 217-2-3 can have an architecture that includes a dielectric metal oxide 222 for serving as a switching medium (e.g., material), and a conductive path can be formed through the switching medium by applying the specific voltage potential 219. The ReRAM cell 217-2-3 architecture can include a bottom electrode 224 (e.g., which can be the same as or different from wire 215-3) coupled to a ground component 225 to serve as a drain. The resistive switching mechanism of the dielectric metal oxide 222 can be based on, for example, forming filaments in the switching medium when a specific voltage potential 219 is applied between the top electrode 220 and the bottom electrode 224. There can be different mechanisms for implementing ReRAM based on different switching materials and / or memory cell configurations.

[0036] Crossbar ReRAM technology can use a silicon-based switching material as the medium for metal filament formation. When a specific voltage potential 219 is applied between two electrodes 220 and 224, nanofilaments can be formed. Since the resistive switching mechanism can be based on an electric field, the memory state (e.g., data value) of the crossbar ReRAM cell can be stable (e.g., capable of withstanding temperature fluctuations from -40 to +125 degrees Celsius, at least one million write / read / erase cycles, and / or providing 10-year data retention at +85 degrees Celsius).

[0037] Crossbar ReRAM technology can be formed in a two-dimensional (2D) architecture (e.g., as shown in Figure 2 ), and / or in a 3D architecture. The 2D architecture can be formed on a single chip (e.g., die), and / or the 3D architecture can be stacked on a single chip to provide petabytes of data storage. The complementary metal-oxide-semiconductor (CMOS) compatibility of crossbar ReRAM technology can enable the integration of both logic (e.g., data processing) and memory (e.g., storage device) onto a single chip. The crossbar ReRAM array can be formed in a one-transistor / one-capacitor (1T1C) configuration and / or in a configuration of one transistor driving n resistive memory cells (1TNR), and other possible configurations.

[0038] Multiple inorganic and organic material systems can achieve thermal and / or ionic resistive switching. Such systems can be used in the in-memory image processors described herein. In multiple embodiments, such systems can include: phase-change chalcogenides (e.g., Ge2 Sb 2 Te 5 such as AgInSbTe; binary transition metal oxides (e.g., NiO, TiO 2 etc.); perovskites (e.g., Sr(Zr)TiO 3 such as PCMO); solid state electrolytes (e.g., GeS, GeSe, SiO x , Cu 2 S etc.); organic charge transfer complexes (e.g., Cu tetracyanoquinodimethane (TCNQ) etc.); organic charge acceptor systems (e.g., Al amino-dicyanoimidazole (AIDCN) etc.); and / or 2D (laminated) insulating materials (e.g., hexagonal BN etc.); and other possible systems for resistive switching.

[0039] Figure 3 Illustrate an example 330 of a parameter matrix 333 used by an image processor 104 and / or an MMU 106 formed in a memory cell array according to multiple embodiments of the present disclosure. As described in connection with Figure 2 A single MMU 206 can be formed as a 3×3 array of resistive ReRAM cells 217 in a RAM array 105 of memory cells in a crossbar switch 211 configuration.

[0040] Each MMU in multiple MMUs in a pipeline (e.g., as shown and described in connection with Figure 5 ) and / or a single MMU that will be reused (e.g., as shown and described in connection with Figure 6 ) can contain (e.g., store) one parameter matrix 333 at a time. One parameter matrix 333 can be selected and / or input from multiple optional parameter matrices. In connection with Figure 4 Examples of multiple different parameter matrices that enable corresponding numbers of multiple different color correction operations to be performed on the MMU 206 of the image processor 104 are described. Multiple optional parameter matrices for implementing multiple different color correction operations can be stored by parameter matrix components (e.g., as shown and described in connection with Figure 6 ) In multiple embodiments, one of the multiple optional parameter matrices can be selected by a control unit (e.g., as shown and described in connection with Figure 6 ) and input into a single MMU 106. One parameter matrix 333 can contain nine (3×3) parameter entries that will be stored on a corresponding array of nine (3×3) ReRAM memory cells 217 of the MMU 206.

[0041] This parameter matrix 333 can be represented by:

[0042]

[0043] Each of the w values may represent a color correction coefficient to be stored at the location of the corresponding ReRAM memory cell 217, as indicated by the subscript of the coefficient w xx Each of the plurality of input vectors 331 used in performing the matrix multiplication with the parameter matrix 333 may be a 3×1 vector 332. In multiple embodiments, the 3×1 vector 332 may have three values (a 1 , a 2 , a 3 ), which correspond to the levels sensed by the sensor circuit 102 for each of the color components (e.g., RGB, CMY, etc.) of the pixel. The output vectors 334 determined by multiplying each input vector 331 with a specific parameter matrix 333 may each also be a 3×1 vector 336 having three values (b 1 , b 2 , b 3 ), which correspond to the color correction levels of the color components (e.g., RGB, CMY, YUV, etc.) of the pixel.

[0044] Thus, the number of parameter entries in a specific parameter matrix 333 is the square of the number of bits in the plurality of input vectors 331 (e.g., 3×3 = 9 parameter entries). The specific parameter entries of the parameter matrix may be stored in the corresponding memory cells 217 in the MMU 206 of the image processor 104, and the number of the plurality of memory cells in the MMU 206 may be at least partially based on the number of bits in the plurality of input vectors (e.g., the nine ReRAM memory cells 217 in the MMU 206 shown and described in conjunction with Figure 2 . The number of the plurality of memory cells in the MMU (e.g., nine) may be arranged in a substantially linear configuration. For example, the edges of each of the two linear directions of the MMU206 may be formed to include the number of the plurality of memory cells 217 corresponding to the number of bits in the plurality of input vectors 331 (e.g., three memory cells 217 correspond to the three bits of the 3×1 vector 332).

[0045] Multiple alternative color spaces may be used. In multiple embodiments, this alternative color space may utilize more than three color components per pixel (e.g., the four-color CMYK color space that adds black (K) to CMY, but the embodiments are not so limited). Thus, the number of parameter entries in a particular parameter matrix 333 may be the square of the number of color components per pixel, which may correspond to a plurality of bits in the input vector 331 (e.g., 4x4 = 16 parameter entries). In multiple embodiments, the number of parameter entries in the parameter matrix may be based on different numbers of bits on each side of the matrix (e.g., 4x3 = 12 parameter bits). A controller (e.g., as shown at 107) may be configured to include a variable number of memory units 217 corresponding to the number of bits of the input vector 331 (e.g., based on the number of color components, the number of bits per pixel, and / or the number of parameter entries in the parameter matrix).

[0046] Figure 4 is a block diagram of an example of multiple color correction operations 440 that may be performed by multiple MMUs 206 of an image processor 104 formed in a memory cell array according to multiple embodiments of the present disclosure.

[0047] The ISP may be an important component of, for example, a digital camera and may be used in various applications (e.g., smart phones, security monitoring systems, and autonomous (self-driving) vehicles, as well as multiple other applications). As described herein, the image processor 104 may perform at least some of the operations that the ISP can perform in memory. For example, the following seven signal processing operations for color correction may be performed by the image processor 104 in memory: defect correction; demosaicking (color interpolation); white balance; color adjustment; gamma adjustment for brightness and / or contrast enhancement; color conversion; and / or downsampling. These operations may be performed in sequence, but the order between some operations (e.g., color interpolation and white balance) may be reversed. This sequence of seven operations for color correction operations is given by way of example herein. However, in multiple embodiments, each such sequence may include fewer than seven or more than seven color correction operations to be performed by the image processor 104.

[0048] As described herein, multiple bits 103 of an input vector 431 can be input to a specific MMU 206 that stores a specific parameter matrix 333 to implement matrix multiplication for a specific color correction operation 440 to be performed by an image processor 104. The input vector 431 can be input (e.g., directly) from a sensor circuit 102 to the specific MMU 206 to perform a first operation in a sequence of color correction operations 440. The input vector 431 input to the specific (e.g., first) MMU 206 can be raw data conforming to a color pattern (e.g., in the form of bits 103). An example of such a color pattern is a Bayer pattern, where half of the total number of pixels are G, and one quarter of the total number of pixels is assigned to both R and B. The Bayer pattern of a color image sensor 102 (e.g., a corresponding digital image sensor and / or light sensor arranged as a color filter array) can be covered with R, G, or B filters arranged in a repeating 2x2 pattern.

[0049] A sequence of multiple color correction operations described herein can be performed by one or more MMUs 206 of an image processor 104. A crossbar array 211 of memory cells (including the MMUs 206 of the image processor 104) is formed as a ReRAM with memory cells 217 configured to store data values in memory and perform color correction operations thereon without feedback from subsequent in-memory color correction operations performed sequentially. For example, an output vector 334 resulting from the execution of one color correction operation can be input as an input vector 331 for performing the next color correction operation in the sequence, but not as an input for the previous color correction operation in the sequence.

[0050] A first color correction operation operating sequentially through the image processor 104 and / or MMU 206 can be a defect correction operation as shown at 442 in Figure 4 for a pixel having a color value different from adjacent pixels. For example, when an image sensor of the sensor circuit 102 senses a pixel having a color value significantly different from its neighbors, the different pixel may distract the viewer and / or be unacceptable to the viewer. Such pixels can be referred to as "defective" and, if not corrected, may appear as errors similar to confetti (e.g., before or after performing subsequent color interpolation operations). These defective pixels can be corrected (e.g., estimated) by interpolating data accurately recorded in their vicinity (e.g., using methods such as median filtering, mean filtering, simple addition, and / or shifting).

[0051] The second color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a color interpolation (demosaicing) operation as shown at 444, whereby the pixel interpolates multiple missing color values from adjacent pixels. The color interpolation operation 444 may use the output vector from the defect correction operation 442 as input 443 to effectuate the performance of the color interpolation operation 444. For example, the color interpolation operation 444 may be performed by processing to interpolate two missing color values for each pixel (e.g., in a Bayer pattern) using one or more of a plurality of possible correction techniques. Such correction techniques may include: bilinear interpolation; median interpolation; bilinear interpolation corrected with gradients; Kodak basic reconstruction; edge-aware interpolation with smooth tone transitions; Kodak edge strength algorithm; variable amounts of gradients; pattern recognition; and / or interpolation by color correction algorithms; and other possibilities.

[0052] The third color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a white balance operation as shown at 446 to shift the color values of multiple color components of a pixel towards whiteness. The white balance operation 446 may use the output vector from the color interpolation 444 as input 445 to effectuate the performance of the white balance operation 446. For example, the white balance operation 446 may be based on the human visual system, which has the ability to map white to a sense of whiteness even though an object may have different radiation when illuminated by different light sources. For example, if a white card is taken outside and exposed to sunlight, the card may appear white to a person. If the white card is placed under a fluorescent light, the card still appears white to a person. If the white card is switched to be illuminated by an incandescent bulb, the card still appears white to a person. Even if the white card is illuminated by a yellow bulb, the card still appears white to a person within a few minutes. When illuminated by each of these light sources, the white card reflects different chromatograms. However, even when perceiving each different chromatogram, the human visual system is still able to make the card appear white.

[0053] In the white balance operation 446 performed by the MMU 206, this may be achieved through white balance processing. The processing of the white balance operation 446 may be performed by shifting, for example, RGB values using the following equation (1):

[0054]

[0055] where W r 、W g and W b are the coefficients in the parameter matrix for shifting the R d 、G d and B dThe original value of R w , G w and B w is a 3×1 vector 336 of RGB values output after performing the white balance operation 446. The balance coefficients W r , W g and W b may depend on lighting conditions (e.g., temperature), the system, and / or the environment, and thus may be adjustable.

[0056] The fourth color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a color adjustment operation as shown at 448 to shift the output of the color values from the sensor circuit to correspond to the human visual perception of the image pixels. The color adjustment operation 448 may use the output vector from the white balance operation 446 as the input 447 to implement the execution of the color adjustment operation 448. For example, the color adjustment operation 448 may be performed because the response of the color filter array used on, for example, a CMOS sensor circuit may not match the response of the human visual system closely enough.

[0057] The color adjustment values for RGB may be determined by performing the following equation (2):

[0058]

[0059] where is a color adjustment coefficient parameter matrix containing coefficients that can be used to perform RGB color adjustment. The values of the color adjustment coefficients may be determined according to different lighting conditions (e.g., temperature), the system, and / or the environment, and thus may be adjustable.

[0060] The fifth color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a gamma / brightness / contrast adjustment operation as shown at 450 to adjust the brightness and / or contrast of multiple pixels in the image by adjusting a vector (e.g., the gamma parameter). The gamma / brightness / contrast adjustment operation 450 may use the output vector from the color adjustment operation 448 as the input 449 to implement the execution of the gamma / brightness / contrast adjustment operation 450. For example, the gamma / brightness / contrast adjustment operation 450 may be used to control the overall brightness (e.g., lightness) and / or contrast of the image. An image with the brightness and contrast not adjusted correctly may appear bleached and / or too dark. Changing the value of the gamma parameter may not only help adjust the brightness but also help adjust the color ratio (e.g., of R to G to B) that affects the contrast.

[0061] Similar to white balance and color adjustment, the gamma / brightness / contrast adjustment may be determined by performing the following formula (3):

[0062]

[0063] wherein is a luminance and contrast coefficient parameter matrix, and are gamma parameter coefficients that can be used together to perform luminance and / or contrast adjustment. The values of the luminance and contrast coefficients and / or the gamma parameter coefficients can be determined according to different lighting conditions (e.g., temperature), systems, and / or environments, and thus can be adjustable.

[0064] The sixth color correction operation sequentially performed by the image processor 104 and / or the MMU 206 can be a color conversion operation as shown at 452, to convert the number of color components of the image pixels provided by the sensor circuit into a corresponding number of color components in a different color space that may be more suitable for further processing, for example. The color conversion operation 452 can use the output vector from the gamma / luminance / contrast adjustment operation 450 as the input 451 to implement the execution of the color conversion operation 452.

[0065] The color conversion operation 452 can use, for example, the three colors (e.g., RGB) of each pixel to be converted into the YUV format (e.g., color space) for further application processing (e.g., video compression and other such applications). This conversion can be achieved by executing Equation (4):

[0066] Y = 0.29900R + 0.58700G + 0.11400B

[0067] Cb = -0.16874R - 0.33126G + 0.50000B + 2 SP / 2 (4)

[0069] Cr = 0.50000R - 0.41869G - 0.08131B + 2 SP / 2

[0070] where SP corresponds to the sample precision. By executing Equation (5), the components of the YUV color space can be converted back to the RGB color space of each pixel:

[0071] R = Y + 1.40200Cr

[0072] G = Y - 0.34414(Cb - 2 SP / 2 ) - 0.71414(Cr - 2 SP / 2 ) (5)

[0074] B = Y + 1.72200(Cb - 2 SP / 2 )

[0075] Using formulas 4 to 5, RGB values can be converted into YCbCr values in the YUV color space that can be used for further (e.g., digital) processing, and the YCbCr values can be converted back to RGB values.

[0076] The seventh color correction operation sequentially performed by the image processor 104 and / or the MMU 206 can be a downsampling operation as shown at 454 to reduce the color values of the color values including the image pixels to a lower number of color values for at least one of the multiple color components in different color spaces. The downsampling operation 454 can use the output vector from the color conversion operation 452 as the input 453 to implement the execution of the downsampling operation 454.

[0077] For example, the downsampling operation 454 can be performed corresponding to the visual perception of the image being processed. One downsampling rate that can be used for this downsampling is "4:2:0", which means that the four pixels of the Y component are not downsampled, so all four pixels are retained, while the Cb component and the Cr component are downsampled 2:1 vertically and horizontally, so only one of the four pixels is retained. In the case of selecting the downsampling rate, there are various techniques to implement the processing of the downsampling operation 454. One technique is to simply copy the value of one pixel and skip the other adjacent pixels. This can reduce complexity but may produce blocking artifacts. Another technique is to apply a filtering algorithm. A compromise technique is to apply averaging between adjacent and / or neighboring pixels.

[0078] In multiple embodiments, further processing 456 can be performed in a sequence of color correction operations 440 by the image processor 104 and / or the MMU 206, and / or can be performed downstream by other components and / or processors (e.g., of a host (not shown)) of the resource implementation 100 illustrated in Figure 1 The further processing 456 can use the output vector from the downsampling operation 454 as the input 455 or start using the output vector from the downsampling operation 454 as the input 455 to implement the execution of the further processing 456. Performing the further processing 456 can produce an output 434 (e.g., an additional 3×1 vector 336 and / or a complete image based on the color-corrected pixels described herein). Examples of this further processing 456 are described in Figure 7 and elsewhere in this document.

[0079] When a digital signal processor (DSP) and / or an application specific integrated circuit (ASIC) external to the memory array in which pixels are stored (e.g., located in and / or associated with a host and / or host processor) is used to perform processing of color correction operations, it may increase the computational complexity and / or memory access. Such external DSPs and / or ASICs may be used, for example, in the architecture of a digital camera, where an application processor is also used to perform digital processing of applications other than image signal and / or color correction processing. However, with respect to image signal and / or color correction processing performed external to the MMU 206 and / or the image processor 104, multiple parameters and / or parameter matrices applied at various positions when performing a sequence of color correction operations may help reduce power consumption and / or increase data access bandwidth. The result of performing such processing can be adjusted by adjusting the values for these parameters and / or parameter matrices, which may vary with different lighting conditions, environments, and / or sensor systems.

[0080] Figure 5 is a block diagram of an example of multiple color correction operations that may be performed by corresponding multiple MMUs 506 of an embodiment of an image processor 504-1 formed in a memory cell array according to embodiments of the present disclosure. The multiple MMUs 506 may be formed in a pipeline configuration 560 (e.g., where each MMU except the last one is coupled to the next MMU). Figure 5 The embodiment of the image processor 504-1 illustrated in may be formed in a memory cell array (e.g., in combination with Figure 2 the crossbar switch array 211 described) as multiple serially coupled MMUs (e.g., as shown at 506-1, 506-2, ……, 506-N). The number of multiple serially coupled MMUs 506 may depend on the corresponding multiple color correction operations (e.g., as shown at 442, 444, ……, 454 and in combination with Figure 4 described) performed in sequence (e.g., as shown at 440 and in combination with Figure 4 described), which may vary depending on a particular implementation.

[0081] The image processor 504-1 may be configured to (e.g., directly) couple to a color filter array (e.g., the sensor circuit 102) to receive multiple bits 503 (e.g., of the input vector 531). The image processor 504-1 may further be configured to perform one of a sequence of multiple color correction operations on each of the corresponding multiple serially coupled MMUs. Thus, the image processor 504-1 may be configured to perform matrix multiplication on the input vector 531 and a parameter matrix on each of the multiple serially coupled MMUs 506 (e.g., as shown at 333 and in combination with Figure 3 described and in combination with Equations 1-3 and Figure 4to determine the color-corrected output vector 534.

[0082] Each of the plurality of input vectors may correspond to an output vector determined by performing a previous in-memory color correction operation on the MMU. For example, as shown and described in connection with Figure 4 the output vector for the defect correction operation 442 generated by the operation of the MMU 506-1 in Figure 5 may correspond to the input vector 443 for performing the color interpolation operation 444, and may operate on the input vector to perform the color interpolation operation by the MMU 506-2 in Figure 5 the color interpolation operation.

[0083] Accordingly, a plurality of serially coupled MMUs may be formed in the pipeline 560 such that the first MMU of the pipeline (e.g., 506-1) may be configured to perform the first color correction operation in the sequence on the first input vector, and the second MMU of the pipeline 560 (e.g., 506-2) may be configured to perform the second color correction operation in the sequence on the first output vector received from the first MMU (e.g., 506-1) as the second input vector of the second MMU (e.g., 506-2). In multiple embodiments, in the same clock cycle in which the second color correction operation is performed on the second MMU (e.g., 506-2), the first MMU (e.g., 506-1) may be configured to repeatedly perform the first color correction operation in the sequence on the third input vector (e.g., different multiple bits 503 of the input vector 531).

[0084] The last output vector (e.g., 534) provided by the last MMU (e.g., 506-N) of the pipeline 560, which is output as the color-corrected output vector, enables the first input vector (e.g., the just-mentioned third input vector among the different multiple bits 503 corresponding to the input vector 531) to be input to the first MMU (e.g., 506-1) of the pipeline 560, and the input and output vectors are continuously moved through the sequence of multiple color correction operations. The color-corrected output vector 534 may be output from the plurality of serially coupled MMUs (e.g., 506-1, 506-2,..., 506-N) for storage (e.g., by other memory cells of the crossbar switch array 211) and / or for further processing (e.g., as shown and described in connection with Figure 4 and 7 described).

[0085] Each of the plurality of input vectors and a particular parameter matrix may be, for example, as shown at 333 and in connection with Figure 3 described and in connection with Formulas 1-3 and Figure 4Description) performs matrix multiplication, and / or can perform specific mathematical operations on an input vector (e.g., as associated with Formulas 4-5 and Figure 4 ) by a corresponding MMU. Specific parameter matrices and / or codes for performing specific mathematical operations can be stored by a specific MMU to implement performing a specific one of multiple color correction operations until a final color-corrected output vector is determined for a first image pixel. Additionally, specific parameter matrices and / or codes for performing specific mathematical operations can be stored by a specific MMU (e.g., each of the pipelines 560 of MMUs 506-1, 506-2, ……, 506-N) to enable repeated input of different multiple-bit 503 input vectors 531 corresponding to different pixels into a first MMU 506-1 and output 534 from a last MMU 506-N. In multiple embodiments, each of the number of multiple output vectors for performing matrix multiplication and the corresponding number of multiple input vectors can be a 3×1 vector. The determined output vector 534 for color correction can also be a 3×1 vector.

[0086] Figure 6 is a block diagram illustrating a configuration example for performing multiple color correction operations 670 on a single MMU 606 of an image processor 604-2 formed in a memory cell array according to multiple embodiments of the present disclosure. Figure 6 The embodiment of the image processor 604-2 illustrated in Figure 2 can be formed in association with a memory cell array (e.g., a crossbar switch array 211 as described in conjunction with

[0087] to include a single MMU 606. The image processor 604-2 can be configured to be (e.g., directly) coupled to a digital image sensor (e.g., sensor circuit 102) to receive (e.g., multiple bits 603 of an input vector 631). Figure 3 The image processor 604-2 can be formed to include parameter matrix components 672, which are configured to store multiple optional parameter matrices (e.g., as shown at 333 and described in conjunction with Figure 4 and described in conjunction with Formulas 1-3 and

[0088] In multiple embodiments, the image processor 604-2 can be configured to be coupled to multiple sensors to receive data values 603 / 631 indicating each level of multiple color components and perform a sequence of multiple color correction operations on a single MMU 606 (as shown at 440 and described in conjunction with Figure 4Description). The image processor 604-2 may execute a sequence of operations to include performing matrix multiplication on an input vector and a selected parameter matrix input by the control unit 671 from the parameter matrix component 672 to a single MMU 606 to determine an output vector 673 for color correction. An adjustment vector (e.g., such as the gamma parameter described in connection with the fifth color correction operation 450 and Figure 4 described) may be input by the control unit 671 to the single MMU 606 (e.g., as another 3×1 parameter to be used in the matrix multiplication of a 3×1 input vector and a selected 3×3 parameter matrix).

[0089] The single MMU 606 of the image processor 604-2 may be formed as part of an array of memory cells (e.g., as described in connection with Figure 2 described) and may be (e.g., directly) coupled to a plurality of sensors. The parameter matrix component 672 and / or the control unit 671 of the image processor 604-2 may each be directly coupled to the single MMU 606 (e.g., as shown at 678 for the parameter matrix component 672 and at 673 and 675 for the control unit 671). The control unit 671 may be directly coupled 677 to the parameter matrix component 672 such that color correction processing is performed on data values stored on the memory cells of the image processor 604-2, and in multiple embodiments, data values related to in-memory color correction processing may not be moved to a host component (not shown) indirectly coupled to the array for processing.

[0090] Each of the plurality of selectable parameter matrices stored by the parameter matrix component 672 and selectable by the control unit 671 for input to the single MMU 606 may include parameter entries configured to implement a particular color correction operation of the execution sequence. The control unit 671 may be configured to direct the single MMU 606 to output an output vector 673 corresponding to a first pixel, the output vector determined by a first color correction operation in the execution sequence for storage by the control unit 671. The control unit 671 may be configured to direct the input 678 from the parameter matrix component 672 to the single MMU 606 of a second selected parameter matrix corresponding to a second color correction operation. The control unit 671 may be configured to direct the input 675 of the stored output vector corresponding to the first pixel as the input vector for performing the second color correction operation. Thus, a sequence of multiple color correction operations may be performed on the single MMU 606 until the expected (e.g., desired) color-corrected output vector for the first pixel is determined. Thus, color correction operations may be performed by the image processor 604-2 on other pixels.

[0091] Figure 7FIG. 780 is a flow diagram of an example of a method of performing color correction by an image processor formed in a memory cell array in accordance with the description of various embodiments of the present disclosure. Unless explicitly stated, the elements of the methods described herein are not bound by a particular order or sequence. Additionally, the various method embodiments or elements thereof described herein may be performed at the same point in time or at substantially the same point in time.

[0092] At block 782, in various embodiments, method 780 may include transmitting an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel (e.g., as described in conjunction with Figure 1 and elsewhere herein) to an image processor formed in a memory cell array. At block 784, in various embodiments, method 780 may include performing a matrix multiplication by the image processor on the input vector and a parameter matrix. At block 786, in various embodiments, method 780 may include determining, by the image processor, an output vector including a plurality of potentially color-corrected components (e.g., the color components may be corrected by performing a color correction operation, as described in conjunction with Figure 3 and 4 and elsewhere herein) based at least in part on the matrix multiplication.

[0093] In various embodiments, method 780 may further include forming an image processor in the array to include an MMU formed as a 3×3 array of ReRAM cells in a crossbar switch configuration. Method 780 may further include forming a 3×3 crossbar switch configuration to correspond to a 3×3 configuration of parameter entries in the parameter matrix (e.g., as described in conjunction with Figure 3 and 4 and elsewhere herein). As described, an alternative number of memory cells may be formed as an MMU in a crossbar switch configuration (e.g., at least in part based on a variable number of bits of the input vector, a variable number of color components, a variable number of bits per pixel, and / or a variable number of parameter entries in the parameter matrix).

[0094] Method 780 may further include performing a sequence of a plurality of color correction operations on the image processor (e.g., as described in conjunction with Figure 4 and elsewhere herein). In various embodiments, the sequence may include performing color correction operations to (e.g., sequentially) include: a defect correction operation on a pixel having a color value significantly different from adjacent pixels; a color interpolation operation for interpolating a plurality of missing color values for a pixel from adjacent pixels; a white balance operation for shifting the color values of the plurality of color components of a pixel towards whiteness; and / or a color adjustment operation for shifting the output of the color values from a sensor circuit to correspond to the perception of an image pixel by human vision (e.g., as described in conjunction with Figure 4 ).

[0095] In multiple embodiments, method 780 may further include outputting a color-corrected output vector or multiple such output vectors from the image processor for: storage by memory units other than the memory units included in the image processor in the array; further processing (e.g., video compression and other such applications) to form a series of images for video presentation; further processing to form a single image for static presentation; further processing in an image recognition operation; and / or further processing by a host processor indirectly coupled to the array including the image processor.

[0096] Method 780 may include, in addition to outputting a color-corrected output vector, performing a sequence of multiple color correction operations on the image processor to further include, in multiple embodiments, at least one of the following: a gamma / brightness / contrast adjustment operation for shifting the brightness and contrast of multiple pixels in an image by adjusting a gamma parameter; a color conversion operation for converting the number of multiple color components of image pixels provided by a sensor circuit into a corresponding number of multiple color components in a different color space more suitable for further processing; and / or a downsampling operation for reducing the color values of image pixels included for at least one of the multiple color components in a different color space to a lower number of color values (e.g., as described in connection with Figure 4 and elsewhere herein).

[0097] In the foregoing detailed description of the present disclosure, reference is made to the accompanying drawings, which form a part hereof, and in which are shown, by way of illustration, one or more embodiments by which the present disclosure may be practiced. The embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the embodiments of the present disclosure, and it is to be understood that other embodiments may be utilized and that process, electrical, and / or structural changes may be made without departing from the scope of the present disclosure.

[0098] As used herein, particularly with respect to the drawings, reference numerals with hyphens and / or designators, such as "M", "N", "X", "Y", etc. (e.g., Figure 5 506-1, 506-2, ……, 506-N in

[0099] It should also be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used herein, unless the context clearly dictates otherwise, the singular forms "a" and "the" include singular and plural referents, such as "a plurality," "at least one," and "one or more" (e.g., a plurality of memory arrays may refer to one or more memory arrays), and "a plurality" is intended to refer to more than one such thing. Further, throughout this application, the word "may" is used in a permissive sense, i.e., having the potential to, rather than in a mandatory sense, i.e., must. The term "comprising" and its derivatives mean "including but not limited to." The term "coupled / coupling" means directly or indirectly connected physically for access and / or for movement (transfer) of instructions (e.g., control signals, address signals, etc.) and data as appropriate in the context. The terms "data" and "data value" may be used interchangeably herein and may have the same meaning as appropriate in the context (e.g., one or more data units or "bits").

[0100] Although example embodiments applicable to a color correction processor have been illustrated and described herein, including combinations and configurations of memory resources, processing resources, a color correction processor (CPU), a matrix multiplication unit (MMU), ReRAM cells, a crossbar array, a parameter matrix, an input vector, an output vector, a controller, a control unit, and parameter matrix components, and other components, embodiments of the present disclosure are not limited to those combinations explicitly recited herein. Other combinations and configurations of memory resources, processing resources, a CPU, an MMU, ReRAM cells, a crossbar array, a parameter matrix, an input vector, an output vector, a controller, a control unit, and parameter matrix components applicable to the color correction processor disclosed herein are precisely included within the scope of the present disclosure.

[0101] Although specific embodiments have been illustrated and described herein, those of ordinary skill in the art will appreciate that arrangements calculated to achieve the same results may replace the specific embodiments shown. The present disclosure is intended to cover modifications or variations of one or more embodiments of the present disclosure. It should be understood that the above description is in an illustrative rather than a limiting sense. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon review of the above description. The scope of one or more embodiments of the present disclosure includes other applications using the above structures and processes. Accordingly, the scope of one or more embodiments of the present disclosure should be determined with reference to the appended claims along with the full scope of equivalents to which such claims are entitled.

[0102] In the foregoing detailed description, for the purpose of simplifying the present disclosure, some features are grouped together in a single embodiment. This method of the present disclosure should not be construed as reflecting an intention that the disclosed embodiments of the present disclosure must use more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the subject matter of the present invention lies in less than all of the features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the detailed description, where each claim stands on its own as a separate embodiment.

Claims

1. A device, wherein include: processor; and a memory device external to the processor and comprising an internal image processor, wherein the internal image processor is configured to: receiving an input vector corresponding to an image pixel from a sensor circuit; and executing a plurality of image signal processing stages that generate respective output vectors corresponding to pixels of the image, The plurality of image signal processing stages include: Defect correction phase; a color interpolation stage following the defect correction stage; and The white balance stage after the defect correction stage, Wherein the plurality of image signal processing stages are performed on the memory device without moving image data corresponding to the image pixels from the memory device to the processor.

2. The apparatus of claim 1, wherein the internal image processor comprises a plurality of matrix multiplication units (MMUs) corresponding to the respective plurality of image processing stages.

3. The apparatus of claim 2, wherein the memory device is a random access memory (RAM) device comprising an array of RAM cells.

4. The apparatus of claim 3, wherein the plurality of MMUs comprises RAM cells of the array.

5. The apparatus of claim 1, wherein the processor is a host processor.

6. The apparatus of claim 1, wherein the plurality of image signal processing stages include: A color adjustment stage after the white balance stage; A color conversion stage after the color adjustment stage; and The color conversion stage is followed by a downsampling stage.

7. The apparatus of claim 6, wherein the plurality of image signal processing stages includes a gamma / brightness / contrast adjustment stage between the color adjustment stage and the color conversion stage.

8. The apparatus of claim 1 , wherein the memory device comprises an array of memory cells configured to store parameter matrices for performing the plurality of image signal processing stages, and wherein the internal image processor is configured to perform the plurality of image signal processing stages without moving parameter matrix data from the memory device to the processor.

9. The apparatus of claim 1, wherein the plurality of image signal processing stages comprises at least seven stages.

10. A method, wherein include: receiving, at a memory device and from a sensor circuit, an input vector corresponding to an image pixel; and executing a plurality of image signal processing stages that generate respective output vectors corresponding to pixels of the image, The performing of the plurality of image signal processing stages comprises: Perform defect correction phase; performing a color interpolation stage after the defect correction stage; and A white balancing phase is performed after the defect correction phase, Wherein the plurality of image signal processing stages are performed on the memory device via an image processor internal to the memory device without moving image data corresponding to the image pixels from the memory device to a processor external to the memory device.

11. The method of claim 10, wherein the memory device comprises an array of memory cells, and wherein the method comprises performing the plurality of image signal processing stages by using a set of memory cells of the array as a matrix multiplication unit.

12. The method according to claim 11, further comprising: include: storing in the array parameter matrix data associated with the image signal processing; and The plurality of image signal processing stages are performed without moving the parameter matrix data from the memory device to the processor external to the memory device.

13. The method of claim 10, wherein performing the plurality of image signal processing stages include: performing a color adjustment phase after the white balance phase; performing a color conversion phase after the color adjustment phase; as well as A downsampling stage is performed after the color conversion stage.

14. A device, wherein include: Host processor; and a memory device external to the host processor and comprising an internal image processor, wherein the internal image processor is configured to: storing a parameter matrix associated with image signal processing in a memory cell array of the memory device; receiving an input vector corresponding to an image pixel from a sensor circuit; performing a first stage of a plurality of image signal processing stages that generates a first output vector corresponding to pixels of the image; providing the first output vector to a subsequent stage of the plurality of image signal processing stages; and executing the subsequent stages to generate a second output vector corresponding to the image pixels, Wherein the plurality of image signal processing stages are performed on the memory device without moving data corresponding to the input vector from the memory device to the host processor and without moving data corresponding to the parameter matrix from the memory device to the host processor.

15. The apparatus of claim 14, wherein the internal image processor is configured to perform the first stage of the plurality of image processing stages by performing a matrix multiplication operation between the parameter matrix and the input vector.

16. The apparatus of claim 14, wherein the internal image processor comprises a plurality of matrix multiplication units corresponding to the plurality of image signal processing stages.

17. The apparatus of claim 16, wherein the plurality of matrix multiplication units are comprised of memory cells within the array.

18. The apparatus of claim 14, wherein the internal image processor comprises a matrix multiplication unit configured to perform matrix multiplication using a different parameter matrix at each of the plurality of image signal processing stages.

Images