Image processor formed in memory cell array

Abstract

Devices, systems, and methods related to an image processor formed in a memory cell array are described. An image processor as described herein is configured to reduce complexity and power consumption and / or increase data access bandwidth by performing image processing in the memory cell array relative to image processing performed by a host processor external to the memory array. For example, one device described herein includes: a sensor circuit configured to provide an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel; and an image processor formed in a memory cell array. The image processor is coupled to the sensor circuit to receive the plurality of bits of the input vector. The image processor is configured to perform a color correction operation in the array by performing a matrix multiplication on the input vector and a parameter matrix to determine an output vector that is color corrected.

Description

Technical Field

[0001] The present disclosure relates generally to semiconductor memory and methods, and more particularly to apparatus, systems, and methods for image processors formed in memory cell arrays. Background Art

[0002] Memory resources are typically provided as internal semiconductor integrated circuits in computers or other electronic systems. There are many different types of memory, including volatile and non-volatile memory. Volatile memory may require power to maintain its data (e.g., host data, error data, etc.). Volatile memory may include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), and thyristor random access memory (TRAM), etc. Non-volatile memory can provide persistent data by saving stored data when power is not supplied. Non-volatile memory may include NAND flash memory, NOR flash memory, and resistance variable memory, such as phase change random access memory (PCRAM) and resistive random access memory (ReRAM), ferroelectric random access memory (FeRAM) and magnetoresistive random access memory (MRAM), such as spin torque transfer random access memory (STT RAM), etc. Summary of the invention

[0003] In one aspect, the present disclosure relates to an image processing device, comprising: a sensor circuit configured to provide an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel; an image processor formed in an array of memory cells; and wherein the image processor is: coupled to the sensor circuit to receive the plurality of bits of the input vector; and configured to perform a color correction operation in the array by performing matrix multiplication on the input vector and a parameter matrix to determine an output vector for color correction.

[0004] In another aspect, the present disclosure relates to an image processing system, comprising: a light sensor arranged in a color filter array to digitally capture a color image, wherein the light sensor is configured to provide a plurality of bits corresponding to a plurality of color components of an image pixel; an image processor formed in a memory cell array as a plurality of matrix multiplication units (MMUs) coupled in series; and wherein the image processor is configured to: couple to the color filter array to receive the plurality of bits; perform one of a plurality of color correction operation sequences on each of the corresponding plurality of the serially coupled MMUs; and perform matrix multiplication on an input vector and a parameter matrix on each of the corresponding plurality of the serially coupled MMUs to determine a color-corrected output vector.

[0005] In another aspect, the present disclosure relates to an image processing system comprising: a digital image sensor for digitally acquiring a color image, wherein each of the sensors is configured to provide a data value indicating a level of one of a plurality of color components of an image pixel; an image processor formed in association with a memory cell array to include: a single matrix multiplication unit (MMU); a parameter matrix component configured to store a plurality of selectable parameter matrices; a control unit configured to select from the plurality of selectable parameter matrices and direct input to the single MMU of the selected parameter matrix; and wherein the image processor is configured to: couple to a plurality of the sensors to receive data values ​​indicating each level of the plurality of color components; perform a sequence of a plurality of color correction operations on the single MMU; and perform matrix multiplication on an input vector input by the control unit from the parameter matrix component to the single MMU and the selected parameter matrix to determine an output vector for color correction.

[0006] In another aspect, the present disclosure relates to a method of operating a memory device for image processing, comprising: transmitting an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel to an image processor formed in an array of memory cells; performing, by the image processor formed in the array, matrix multiplication on the input vector and a parameter matrix; and determining, by the image processor formed in the array, an output vector comprising a plurality of color-corrected components based at least in part on the matrix multiplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating an example of resource implementation that may be used for an image processor including one formed in a memory cell array according to various embodiments of the present disclosure.

[0008] Figure 2 is a block diagram of an example of a portion of a resistive random access memory (ReRAM) array operable as an image processor according to various embodiments of the present disclosure.

[0009] Figure 3 Examples of parameter matrices that may be used by an image processor formed in a memory cell array are described in accordance with various embodiments of the present disclosure.

[0010] Figure 4 is a block diagram illustrating examples of multiple color correction operations that may be performed by multiple matrix multiplication units (MMUs) of an image processor formed in a memory cell array according to multiple embodiments of the present disclosure.

[0011] Figure 5is a block diagram illustrating examples of multiple color correction operations that may be performed by corresponding multiple MMUs of an image processor formed in a memory cell array according to multiple embodiments of the present disclosure.

[0012] Figure 6 is a block diagram illustrating a configuration example for performing multiple color correction operations on a single MMU of an image processor formed in a memory cell array according to various embodiments of the present disclosure.

[0013] Figure 7 is a flow chart illustrating an example of an image processor formed in a memory cell array that performs color correction according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] The present disclosure includes systems, apparatus, and methods associated with an image processor formed in a memory cell array. In various embodiments, the apparatus includes a sensor circuit configured to provide an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel; and an image processor formed in the memory cell array. The image processor is coupled to the sensor circuit to receive the plurality of bits of the input vector. The image processor is configured to perform a color correction operation in the memory cell array by performing a matrix multiplication on the input vector and a parameter matrix to determine a color-corrected output vector.

[0015] As the demand for higher image resolution increases, the data access bandwidth from memory (e.g., DRAM) to a processing unit of an image signal processor (ISP) (e.g., on a host that may be indirectly coupled to the memory) is increasing. This increase in data access bandwidth may result in excessive power consumption of such an ISP (e.g., for mobile and / or remote devices such as smartphones).

[0016] The present disclosure describes multiple embodiments of image processors to perform color correction processing in a memory cell array to overcome this potential difficulty and provide other advantages. Together with the absence of data feedback loops and / or related data dependencies between color correction operations, various embodiments of signal processing implementations for color correction described herein can allow most (e.g., all) of such processing (e.g., its computational units and / or control units) to be moved to a memory chip (e.g., associated with and / or within a RAM array) to reduce the complexity of memory accesses.

[0017] In the architecture described herein, an image processor may be formed as part of a RAM (ReRAM) to act as a processor in memory (PIM). In various embodiments, this image processor may be referred to as an in-memory color correction processor. Relative to other ISP implementations, this image processor may reduce power consumption for color correction without causing performance degradation, without adding significantly more components to the memory chip. For example, the crossbar architecture of the ReRAM memory cells may not only store (e.g., for reading and / or writing) data, but may also be used to perform arithmetic operations other than the matrix multiplication described herein, such as addition, subtraction, multiplication, and / or division, on operands via multiple operations without incorporating additional components.

[0018] The present disclosure Figure 2 ReRAM cells and crossbar configurations are shown for computing the color correction operations described herein. For example, a crossbar with a 3×3 ReRAM cell configuration can be used to perform matrix multiplication on a 3×3 matrix (parameter matrix) with a 3×1 vector and then provide the output as a 3×1 vector. As described herein, without modifying the inherent configuration of the ReRAM crossbar array, a crossbar of 3×3=9 ReRAM cells can be used as an MMU to facilitate or replace an ISP.

[0019] According to the combination Figure 5 In the pipelined image processor configuration described above, multiple crossbar switch arrays (e.g., multiple 3×3 MMUs) can be configured to perform a corresponding number of multiple ISP stages of color correction for a corresponding number of multiple pixels simultaneously (e.g., during the same clock cycle). In multiple embodiments (e.g., as combined with Figure 4 and 5 ), the number of multiple MMUs, multiple ISP stages, and multiple pixels can each be 7. For example, raw data of a first pixel (e.g., a 3×1 vector corresponding to RGB color components) can be input to a first crossbar switch for a first stage of processing. The 3×1 vector can be output from the first crossbar switch to be input to a second crossbar switch for a second stage of processing of the first pixel. At the same time, the first crossbar switch can perform the first stage of processing on the second pixel. Continuing this pipeline mode, all seven processing stages for seven different pixels can be performed substantially simultaneously.

[0020] Corresponding to the combination Figure 6In the described image processor configuration, one crossbar switch can be used to perform seven processing stages for each pixel. For example, the first stage (e.g., clock cycle) can be to input one pixel raw data (e.g., RGB vector) into one crossbar switch for the first stage of processing. The second stage of processing can be to input the output from the first stage (e.g., 3×1 vector) back into the same crossbar switch for the second stage of processing on the same pixel. After seven stages, all seven processing stages for the same pixel can be completed on one crossbar switch array.

[0021] Thus, due to the inherent signal processing capabilities of ReRAM, the color correction processing of the ISP can be performed by and / or on the memory. Furthermore, for example, by using a crossbar ReRAM architecture, additional processing components for color correction can be reduced (e.g., unnecessary) because the crossbar of the ReRAM can not only store data, but can also be used to perform matrix multiplications and other arithmetic operations described herein. Thus, the image processor described herein can facilitate reducing power consumption and / or increasing data access bandwidth related to image signal and / or color correction processing.

[0022] The figures herein follow a numbering convention in which the first one or more digits of a reference numeral correspond to the figure number and the remaining digits identify an element or component in the figure. Similar elements or components between different figures may be identified by using similar numerals. For example, 104 may refer to Figure 1 Component "04" in , and similar components in Figure 5 It can be marked as 504.

[0023] Figure 1 is a schematic diagram illustrating an example of resource implementation that may be used for an image processor including one formed in a memory cell array according to various embodiments of the present disclosure. Figure 1 The resource implementation 100 illustrated in FIG. 1 is intended to show components that may be particularly suitable for in-memory color correction processing, and other electrical, mechanical, and / or processing elements that may facilitate such processing may be omitted for clarity.

[0024] The resource implementation 100 described herein includes a sensor circuit 102 configured to provide an input vector as a plurality of bits 103 corresponding to a plurality of color components of an image pixel. In various embodiments, the sensor circuit 102 may be or may include a digital image sensor and / or a light sensor arranged in a color filter array (e.g., a Bayer filter, among other possible configurations) for digital acquisition of a color image. The digital image sensor and / or light sensor may be configured to provide a plurality of bits 103 corresponding to a plurality of color components of an image pixel. For example, each of the sensors may be configured to provide a data value indicating a level of one of the plurality of color components of an image pixel.

[0025] The resource implementation 100 described herein also includes a memory cell array (e.g., as combined with Figure 2 The image processor 104 may be coupled to the sensor circuit 102 (e.g., directly via a bus and without intermediate processing circuitry) to receive a plurality of bits 103 of an input vector corresponding to a plurality of color components of an image pixel sensed and / or acquired by the sensor circuit 102. The image processor 104 may be configured to process the input vector (e.g., as shown and described at 331) by processing the input vector 103. Figure 3 ) and a parameter matrix (e.g., as shown and combined at 333 Figure 3 ) performs a matrix multiplication to perform a color correction operation to determine a color-corrected output vector (e.g., as shown at 334 and in conjunction with Figure 3 and described elsewhere in this document).

[0026] The memory cell array including the image processor 104 may be formed to include at least a portion of the RAM array 105. The image processor 104 may be formed in the RAM array 105 to include the MMU 106. The MMU 106 may be formed as a plurality of memory cells in the array (e.g., as combined with Figure 2 104 and / or MMU 106. The memory cells may be configured to perform (e.g., by matrix multiplication) a determination of a color-corrected output vector. In various embodiments, the determination of the color-corrected output vector may be performed within a memory (e.g., in an array) without the memory cells being coupled to a sensing circuit including a sense amplifier (not shown).

[0027] The number of memory cells of MMU 106 may be formed to correspond to the parameter matrix (e.g., as shown at 333 and in conjunction with Figure 3 331 and in combination with the input vector (e.g., as shown at 331 and in combination with the input vector 332). Figure 3 The plurality of bits 103 (described above) may be three bits to correspond to three color components. The image processor 104 may be configured to perform a matrix multiplication on the input vector and the parameter matrix to generate an output vector (e.g., as shown at 334 and combined with Figure 3 ) to include three color-corrected bits in various embodiments. In various embodiments, a fast Fourier transform (FFT) may be performed for the matrix multiplication.

[0028] The resource implementation 100 described herein may also include a controller 107 configured to direct input of an input vector from the sensor circuit 102 (e.g., directly) to the image processor 104 and / or the MMU 106. The controller 107 may also be configured to direct multiple color correction operations in the array through the image processor 104 (e.g., as combined with Figure 4 ) is executed to determine a color-corrected output vector.

[0029] The resource implementation 100 described herein may also include a display processor 108. The display processor 108 may be configured to perform other ISP operations after the image processor 104 performs the color correction operations to produce an output vector as described herein. In various embodiments, the output from the display processor 108 may be an image and / or a series of images that may be displayed by a screen 109 (e.g., a user interface).

[0030] In some embodiments, the image processor 104 and / or the MMU 106 may be implemented on a plurality of memory resources. As used herein, a "memory resource" is a general term intended to include at least memory (e.g., memory cells) arranged, for example, in a plurality of memory bank groups, memory banks, memory bank segments, sub-arrays, and / or rows of a plurality of memory devices. In various embodiments, the memory resource may be or may include a plurality of volatile memory devices formed and / or operable as RAM, DRAM, SRAM, SDRAM, and / or TRAM, as well as other types of volatile memory devices. Alternatively or in addition, in various embodiments, the memory resource may be or may include a plurality of non-volatile memory devices formed and / or operable as NAND, NOR, PCRAM, ReRAM, FeRAM, MRAM, and / or STT RAM, as well as other types of non-volatile memory devices.

[0031] Figure 2 is a block diagram of an example of a ReRAM array that may be used as part of an image processor 104 and / or MMU 106 according to various embodiments of the present disclosure. In various embodiments, an image processor (e.g., as shown at 104, 504-1, and 604-2 and respectively in conjunction with Figure 1 , 5 and 6) can be formed in a memory cell array (for example, shown at 105 and combined Figure 1 The RAM array described herein is formed in and / or as the memory cell array. Figure 2The portion of the ReRAM array illustrated in FIG. 204 may correspond to an embodiment of a single MMU 206, but the embodiments of the image processor 104 described herein are not limited to a single MMU, the MMU is formed as a ReRAM array, and / or Figure 2 In addition, the multiple ReRAM memory cells used Figure 2 The architecture of the memory cell array of the single MMU 206 shown in FIG. 1 may correspond to the architecture of the remainder of the RAM array 105 that is not used as a plurality of MMUs 206 (eg, the remainder may be used for other data storage and / or processing operations), but embodiments are not so limited.

[0032] like Figure 2 As shown in , in various embodiments, the memory cell array of a single MMU 206 may be formed as a 3×3 array of ReRAM cells in a crossbar configuration 211. Thus, each ReRAM memory cell 217 may be coupled to a first conductive line 213 and a second conductive line 215, and / or formed between the first conductive line 213 and the second conductive line 215. In various embodiments, each of the first conductive lines 213 may correspond to a word line, and each of the second conductive lines 215 may correspond to a bit line, or vice versa. For example, in Figure 2 In the embodiment of the 3×3 crossbar switch array 211 shown in FIG, the MMU 206 includes ReRAM cells 217-1-1, 217-1-2, and 217-1-3 coupled to the conductor 213-1, ReRAM cells 217-2-1, 217-2-2, and 217-2-3 coupled to the conductor 213-2, and ReRAM cells 217-3-1, 217-3-2, and 217-3-3 coupled to the conductor 213-3. Due to the configuration of the 3×3 crossbar switch array 211, ReRAM cells 217-1-1, 217-2-1 and 217-3-1 are coupled to conductor 215-1, ReRAM cells 217-1-2, 217-2-2 and 217-3-2 are coupled to conductor 213-2, and ReRAM cells 217-1-3, 217-2-3 and 217-3-3 are coupled to conductor 215-3.

[0033] An example memory cell included in a crossbar array 211 configuration of ReRAM cells for a single MMU 206 is shown in more detail at 217-2-3 in the array and alongside the array. Details of ReRAM cell 217-2-3 show that in various embodiments, the ReRAM cell may have an architecture including a top electrode 220 (e.g., which may be the same or different than conductive line 213-2) coupled to a source component (not shown) to apply a particular voltage potential 219. The ReRAM cell 217-2-3 may have an architecture including a dielectric metal oxide 222 for acting as a switching medium (e.g., a material) through which a conductive path may be formed via application of a particular voltage potential 219. The ReRAM cell 217-2-3 architecture may include a bottom electrode 224 (e.g., which may be the same or different than conductive line 215-3) coupled to a ground component 225 to act as a drain. The resistive switching mechanism of the dielectric metal oxide 222 may be based on, for example, the formation of filaments in the switching medium when a particular voltage potential 219 is applied between the top electrode 220 and the bottom electrode 224. There may be different mechanisms for implementing ReRAM based on different switching materials and / or memory cell configurations.

[0034] Crossbar ReRAM technology may use silicon-based switching materials as a medium for metal filament formation. When a specific voltage potential 219 is applied between two electrodes 220 and 224, nanofilaments may be formed. Because the resistive switching mechanism may be based on an electric field, the memory state (e.g., data value) of the crossbar ReRAM cell may be stable (e.g., able to withstand temperature fluctuations from -40 to +125 degrees Celsius, at least one million write / read / erase cycles, and / or provide 10 years of data retention at +85 degrees Celsius).

[0035] Crossbar ReRAM technology can be formed in a two-dimensional (2D) architecture (e.g., Figure 2 ) and / or in a 3D architecture. A 2D architecture can be formed on a single chip (e.g., a die) and / or a 3D architecture can be stacked on a single chip to provide terabytes of data storage. The complementary metal oxide semiconductor (CMOS) compatibility of the crossbar ReRAM technology can enable both logic (e.g., data processing) and memory (e.g., storage devices) to be integrated onto a single chip. The crossbar ReRAM array can be formed in a one transistor / one capacitor (1T1C) configuration and / or a configuration with one transistor driving n resistive memory cells (1TNR), among other possible configurations.

[0036] A variety of inorganic and organic material systems can achieve thermal and / or ionic resistance switching. Such systems can be used in the in-memory image processor described herein. In various embodiments, such systems can include: phase change chalcogenides (e.g., Ge2Sb2Te5, AgInSbTe, etc.); binary transition metal oxides (e.g., NiO, TiO2, etc.); perovskites (e.g., Sr(Zr)TiO3, PCMO, etc.); solid electrolytes (e.g., GeS, GeSe, SiO x , Cu2S, etc.); organic charge transfer complexes (e.g., Cu tetracyanoquinodimethane (TCNQ), etc.); organic charge acceptor systems (e.g., Al amino-dicyanoimidazole (AIDCN), etc.); and / or 2D (layered) insulating materials (e.g., hexagonal BN, etc.); and other possible systems for resistive switching.

[0037] Figure 3 An example 330 of a parameter matrix 333 used by the image processor 104 and / or the MMU 106 formed in a memory cell array according to various embodiments of the present disclosure is illustrated. Figure 2 As depicted, a single MMU 206 may be formed in the RAM array 105 of memory cells as a 3×3 array of resistive ReRAM cells 217 in a crossbar 211 configuration.

[0038] For each of the multiple MMUs in the pipeline (e.g., as combined Figure 5 ) and / or a single MMU that will be reused (e.g., as combined with Figure 6 333. A parameter matrix 333 may be selected and / or input from a plurality of selectable parameter matrices. Figure 4 An example of a plurality of different parameter matrices that enable a corresponding number of a plurality of different color correction operations to be performed on the MMU 206 of the image processor 104 is described. A plurality of optional parameter matrices for implementing the plurality of different color correction operations may be stored by the parameter matrix component (e.g., as shown at 672 and in conjunction with Figure 6 In various embodiments, one of a plurality of selectable parameter matrices may be selected by a control unit (e.g., as shown at 671 and in conjunction with Figure 6 Description) selection for input to a single MMU 106. One parameter matrix 333 may include nine (3×3) parameter entries to be stored on corresponding arrays of nine (3×3) ReRAM memory cells 217 of the MMU 206.

[0039] This parameter matrix 333 can be represented as follows:

[0040]

[0041] Each w value may represent a color correction coefficient to be stored at a location in the corresponding ReRAM memory cell 217, as represented by the coefficient w xx The subscripts of are indicated. Each of the plurality of input vectors 331 used in performing the matrix multiplication with the parameter matrix 333 may be a 3×1 vector 332. In various embodiments, the 3×1 vector 332 may have three values ​​(a1, a2, a3) corresponding to the levels sensed by the sensor circuit 102 for each of the color components (e.g., RGB, CMY, etc.) of the pixel. The output vectors 334 determined by the matrix multiplication of each input vector 331 with a particular parameter matrix 333 may also each be a 3×1 vector 336 having three values ​​(b1, b2, b3) corresponding to the color correction levels of the color components (e.g., RGB, CMY, YUV, etc.) of the pixel.

[0042] Thus, the number of parameter entries in a particular parameter matrix 333 is the square of the number of bits in the input vector 331 (e.g., 3×3=9 parameter entries). The particular parameter entries of the parameter matrix may be stored by a corresponding memory unit 217 in the MMU 206 of the image processor 104, and the number of memory units in the MMU 206 may be based at least in part on the number of bits in the input vector (e.g., in combination with Figure 2 206). The number of memory cells in the MMU (e.g., nine) may be arranged in a substantially rectilinear configuration. For example, an edge in each of two rectilinear directions of the MMU 206 may be formed to include a number of memory cells 217 corresponding to the number of bits of the input vector 331 (e.g., three memory cells 217 corresponding to three bits of the 3×1 vector 332).

[0043] Multiple alternative color spaces may be used. In multiple embodiments, such an alternative color space may utilize more than three color components per pixel (e.g., adding black (K) to a four color CMYK color space of CMY, but embodiments are not so limited). Thus, the number of parameter entries in a particular parameter matrix 333 may be the square of the number of color components per pixel, which may correspond to a number of bits in the input vector 331 (e.g., 4x4=16 parameter entries). In multiple embodiments, the number of parameter entries in the parameter matrix may be based on a different number of bits on each side of the matrix (e.g., 4x3=12 parameter bits). The controller (e.g., as shown at 107) may be configured to include a variable number of multiple memory cells 217 to correspond to the number of multiple bits of the input vector 331 (e.g., based on the number of color components, the number of bits per pixel, and / or the number of parameter entries in the parameter matrix).

[0044] Figure 4 is a block diagram illustrating an example of a plurality of color correction operations 440 that may be performed by a plurality of MMUs 206 of an image processor 104 formed in a memory cell array, according to various embodiments of the present disclosure.

[0045] The ISP may be an important component of, for example, a digital camera, and may be used in a variety of applications (e.g., smart phones, security surveillance systems, and autonomous (self-driving) vehicles, among many other applications). As described herein, the image processor 104 may perform, in memory, at least some of the operations that the ISP may perform. For example, the following seven signal processing operations for color correction may be performed in memory by the image processor 104: defect correction; demosaicing (color interpolation); white balance; color adjustment; gamma adjustment for brightness and / or contrast enhancement; color conversion; and / or downsampling. These operations may be performed sequentially, but the order between some operations (e.g., color interpolation and white balance) may be reversed. The seven operations of such a sequence of color correction operations are given herein by way of example. However, in various embodiments, such sequences may each include less than seven or more than seven color correction operations to be performed by the image processor 104.

[0046] As described herein, a plurality of bits 103 of an input vector 431 may be input to a particular MMU 206 storing a particular parameter matrix 333 to implement matrix multiplication to perform a particular color correction operation 440 by the image processor 104. The input vector 431 may be input (e.g., directly) from the sensor circuit 102 to the particular MMU 206 to perform a first operation in a sequence of color correction operations 440. The input vector 431 input to the particular (e.g., first) MMU 206 may be raw data (e.g., in the form of bits 103) that conforms to a color pattern. An example of such a color pattern is a Bayer pattern, where half of the total number of pixels are G, and a quarter of the total number of pixels are allocated to both R and B. The Bayer pattern of a color image sensor 102 (e.g., a corresponding digital image sensor and / or photosensor arranged in a color filter array) may be covered with R, G, or B filters arranged in a repeating 2x2 pattern.

[0047] The sequence of multiple color correction operations described herein may be performed by one or more MMUs 206 of the image processor 104. The crossbar array 211 of memory cells (including the MMUs 206 of the image processor 104) is formed as a ReRAM having memory cells 217 that are configured to store data values ​​in memory and perform color correction operations thereon without feedback from subsequent in-memory color correction operations performed in sequence. For example, an output vector 334 resulting from performing one color correction operation may be input as an input vector 331 for performing the next color correction operation in the sequence, but not as an input for the previous color correction operation in the sequence.

[0048] The first color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be performed for pixels having color values ​​different from those of neighboring pixels, such as in Figure 4 442 in . For example, when an image sensor of sensor circuit 102 senses a pixel having a color value that is significantly different from its neighbors, the different pixels may be distracting and / or unacceptable to a viewer. Such pixels may be referred to as "defective" and, if not corrected, may appear as errors similar to confetti (e.g., before or after performing subsequent color interpolation operations). These defective pixels may be corrected (e.g., estimated) by interpolating data accurately recorded in their vicinity (e.g., using methods such as median filtering, mean filtering, simple addition and / or shifting).

[0049] The second color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a color interpolation (demosaicing) operation as shown at 444, interpolating multiple missing color values ​​from neighboring pixels for a pixel. The color interpolation operation 444 may use the output vector from the defect correction operation 442 as an input 443 to enable performing the color interpolation operation 444. For example, the color interpolation operation 444 may be performed by a process of interpolating two missing color values ​​for each pixel (e.g., in a Bayer pattern) using one or more of a plurality of possible correction techniques. Such correction techniques may include: bilinear interpolation; median interpolation; bilinear interpolation with gradient correction; Kodak basic reconstruction; edge-sensitive interpolation with smooth tonal transitions; Kodak edge intensity algorithm; variable number of gradients; pattern recognition; and / or interpolation by a color correction algorithm; among other possibilities.

[0050] The third color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a white balance operation as shown at 446 to move the color values ​​of the plurality of color components of the pixel toward white. The white balance operation 446 may use the output vector from the color interpolation 444 as an input 445 to enable the execution of the white balance operation 446. For example, the white balance operation 446 may be based on the human visual system, which has the ability to map white to a white perception, even though the object may have different radiation when illuminated by different light sources. For example, if a white card is taken outside and exposed to sunlight, the card may appear white to a person. If the white card is placed under a fluorescent light, the card still appears white to a person. If the white card is switched to be illuminated by an incandescent light bulb, the card still appears white to a person. Even if the white card is illuminated by a yellow light bulb, the card still appears white to a person within a few minutes. When illuminated by each of these light sources, the white card reflects a different color spectrum. Yet, the human visual system is able to make the card appear white even when perceiving each different color spectrum.

[0051] This may be accomplished by a white balance process in a white balance operation 446 performed by the MMU 206. The process of the white balance operation 446 may be performed by shifting, for example, RGB values ​​using the following equation (1):

[0052]

[0053] Where W r , W g and W b are the coefficients in the parameter matrix used to shift the input 445 of the 3×1 vector 332 respectively. d , G d and B d The original value of R w , G w and B w is the 3×1 vector 336 of RGB values ​​output after performing the white balance operation 446. The balance coefficient W r , W g and W b The value of may depend on lighting conditions (eg, temperature), the system and / or the environment and may therefore be adjustable.

[0054] A fourth color correction operation sequentially performed by image processor 104 and / or MMU 206 may be a color adjustment operation as shown at 448 to shift the output of color values ​​from the sensor circuit to correspond to the perception of image pixels by human vision. Color adjustment operation 448 may use the output vector from white balance operation 446 as input 447 to enable performing color adjustment operation 448. For example, color adjustment operation 448 may be performed because the response of a color filter array used on, for example, a CMOS sensor circuit may not match the response of the human visual system closely enough.

[0055] The color adjustment value of RGB can be determined by executing the following equation (2):

[0056]

[0057] in is a color adjustment coefficient parameter matrix including coefficients that can be used to perform RGB color adjustment. The values ​​of the color adjustment coefficients can be determined according to different lighting conditions (eg, temperature), systems and / or environments, and thus can be adjustable.

[0058] A fifth color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a gamma / brightness / contrast adjustment operation as shown at 450 to shift the brightness and / or contrast of a plurality of pixels in the image by adjusting an adjustment vector (e.g., a gamma parameter). The gamma / brightness / contrast adjustment operation 450 may use the output vector from the color adjustment operation 448 as an input 449 to enable the execution of the gamma / brightness / contrast adjustment operation 450. For example, the gamma / brightness / contrast adjustment operation 450 may be used to control the overall brightness (e.g., lightness) and / or contrast of the image. An image for which the brightness and contrast are not adjusted correctly may appear bleached and / or too dark. Changing the value of the gamma parameter may help adjust not only the brightness, but also the color ratios (e.g., R to G to B) that affect the contrast.

[0059] Similar to white balance and color adjustments, gamma / brightness / contrast adjustments can be determined by executing the following equation (3):

[0060]

[0061] in is the brightness and contrast coefficient parameter matrix, and The values ​​of the brightness and contrast coefficients and / or the gamma parameter coefficients may be determined according to different lighting conditions (eg, temperature), systems and / or environments, and thus may be adjustable.

[0062] The sixth color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a color conversion operation as shown at 452 to convert the number of multiple color components of the image pixel provided by the sensor circuit to a corresponding number of multiple color components in a different color space that may be more suitable for further processing, for example. The color conversion operation 452 may use the output vector from the gamma / brightness / contrast adjustment operation 450 as input 451 to implement the color conversion operation 452.

[0063] The color conversion operation 452 may, for example, convert the three colors (e.g., RGB) of each pixel to a YUV format (e.g., color space) for further application processing (e.g., video compression and other such applications). This conversion may be achieved by executing equation (4):

[0064] Y=0.29900R+0.58700G+0.11400B

[0065] Cb=-0.16874R-0.33126G+0.50000B+2 SP / 2 (4)

[0067] Cr=0.50000R-0.41869G-0.08131B+2 SP / 2

[0068] Where SP corresponds to the sample precision. The components of the YUV color space can be converted back to the RGB color space for each pixel by executing formula (5):

[0069] R=Y+1.40200Cr

[0070] G=Y-0.34414(Cb-2 SP / 2 )-0.71414(Cr-2 SP / 2 ) (5)

[0072] B=Y+1.72200(Cb-2 SP / 2 )

[0073] Using Equations 4-5, RGB values ​​may be converted to YCbCr values ​​of a YUV color space that may be used for further (eg, digitization) processing, and YCbCr values ​​may be converted back to RGB values.

[0074] A seventh color correction operation sequentially performed by the image processor 104 and / or the MMU 206 may be a downsampling operation as shown at 454 to reduce the color values ​​comprising the image pixels to a lower number of color values ​​for at least one of the plurality of color components in the different color space. The downsampling operation 454 may use the output vector from the color conversion operation 452 as input 453 to enable performing the downsampling operation 454.

[0075] For example, the downsampling operation 454 may be performed to correspond to the visual perception of the image being processed. One downsampling rate that may be used for this downsampling is "4:2:0", which means that the four pixels of the Y component are not downsampled, so all four pixels are retained, while the Cb component and the Cr component are vertically and horizontally downsampled 2:1, so only one of the four pixels is retained. In the case of selecting a downsampling rate, there are multiple techniques to implement the processing of the downsampling operation 454. One technique is to copy the value of only one pixel and skip other adjacent pixels. This can reduce complexity, but may produce blocking effects. Another technique is to apply a filtering algorithm. A compromise technique is to apply averaging between adjacent and / or neighboring pixels.

[0076] In various embodiments, further processing 456 may be performed by image processor 104 and / or MMU 206 in a sequence of color correction operations 440 and / or may be performed by Figure 1 4. The further processing 456 may use the output vector from the downsampling operation 454 as input 455 or begin using the output vector from the downsampling operation 454 as input 455 to enable performing the further processing 456. Performing the further processing 456 may produce the output 434 (e.g., an additional 3×1 vector 336 and / or a complete image based on the color corrected pixels described herein). Figure 7 Examples of this further processing 456 are described elsewhere herein.

[0077] When a digital signal processor (DSP) and / or an application specific integrated circuit (ASIC) external to the memory array in which the pixels are stored (e.g., located in or associated with a host and / or host processor) is used to implement the processing of the color correction operation, there may be an increase in computational complexity and / or memory access. Such an external DSP and / or ASIC may be used, for example, in the architecture of a digital camera, where an application processor is also used to perform digital processing for applications other than image signal and / or color correction processing. However, multiple parameters and / or parameter matrices applied at various locations when performing a sequence of color correction operations may help reduce power consumption and / or increase data access bandwidth relative to image signal and / or color correction processing performed external to the MMU 206 and / or image processor 104. The results of performing such processing may be adjusted by adjusting the values ​​used for these parameters and / or parameter matrices, which may vary with different lighting conditions, environments, and / or sensor systems.

[0078] Figure 5 5 is a block diagram illustrating examples of multiple color correction operations that may be performed by corresponding multiple MMUs 506 of an embodiment of an image processor 504-1 formed in a memory cell array according to multiple embodiments of the present disclosure. The multiple MMUs 506 may be formed in a pipeline configuration 560 (e.g., where each MMU except the last MMU is coupled to the next MMU). Figure 5 The embodiment of the image processor 504-1 described in the above may be formed in a memory cell array (eg, in combination with Figure 2 , as a plurality of MMUs coupled in series (e.g., as shown at 506-1, 506-2, ..., 506-N). The number of the plurality of MMUs coupled in series 506 may depend on the order (e.g., as shown at 440 and in combination with Figure 4 442, 444, ..., 454 and the like. Figure 4 Description), which may vary depending on the specific implementation.

[0079] The image processor 504-1 may be configured to couple (e.g., directly) to a color filter array (e.g., sensor circuit 102) to receive the plurality of bits 503 (e.g., of an input vector 531). The image processor 504-1 may be further configured to perform one of a plurality of color correction operation sequences on each of the corresponding plurality of serially coupled MMUs. Thus, the image processor 504-1 may be configured to perform a matrix multiplication (e.g., as shown at 333 and in conjunction with) on the input vector 531 and a parameter matrix on each of the plurality of serially coupled MMUs 506. Figure 3 Describe and combine equations 1-3 and Figure 4Description) to determine the color-corrected output vector 534.

[0080] Each of the plurality of input vectors may correspond to an output vector determined by performing a previous in-memory color correction operation on the MMU. Figure 4 Display and description can be done by Figure 5 The output vector for the defect correction operation 442 generated by the MMU 506-1 in the embodiment may correspond to the input vector 443 for performing the color interpolation operation 444, which may be operated on to obtain the color interpolation result by Figure 5 The MMU 506-2 in performs color interpolation operations.

[0081] Thus, a plurality of serially coupled MMUs may be formed in the pipeline 560 such that a first MMU (e.g., 506-1) of the pipeline may be configured to perform a first color correction operation in a sequence on a first input vector, and a second MMU (e.g., 506-2) of the pipeline 560 may be configured to perform a second color correction operation in a sequence on a first output vector received from the first MMU (e.g., 506-1) as a second input vector of the second MMU (e.g., 506-2). In various embodiments, in the same clock cycle with the second color correction operation performed on the second MMU (e.g., 506-2), the first MMU (e.g., 506-1) may be configured to repeatedly perform the first color correction operation in a sequence on a third input vector (e.g., on a different plurality of bits 503 of the input vector 531).

[0082] The last output vector (e.g., 534) provided by the last MMU (e.g., 506-N) of the pipeline 560 is output as a color-corrected output vector from the last MMU (e.g., 506-N) of the pipeline to enable the first input vector (e.g., the third input vector just mentioned in the different plurality of bits 503 corresponding to the input vector 531) to be input to the first MMU (e.g., 506-1) of the pipeline 560, and the input and output vectors are continuously moved through the sequence of the plurality of color correction operations. The color-corrected output vector 534 may be output from the plurality of serially coupled MMUs (e.g., 506-1, 506-2, ..., 506-N) for storage (e.g., by other memory units of the crossbar array 211) and / or for further processing (e.g., as combined with Figure 4 and 7 describe).

[0083] Each of the plurality of input vectors and a particular parameter matrix (eg, as shown and combined at 333) may be Figure 3 Describe and combine formulas 1-3 and Figure 4Description) to perform matrix multiplication, and / or the input vector (e.g., as combined with Formulas 4-5 and Figure 4 ) to perform a specific mathematical operation. A specific parameter matrix and / or code for performing the specific mathematical operation may be stored by a specific MMU to enable performing a specific one of the multiple color correction operations until a last color-corrected output vector is determined for the first image pixel. In addition, a specific parameter matrix and / or code for performing the specific mathematical operation may be stored by a specific MMU (e.g., each of the pipelines 560 of MMUs 506-1, 506-2, ..., 506-N) to enable repeated input of input vectors 531 of different multiple bits 503 corresponding to different pixels to be input to the first MMU 506-1 and output 534 from the last MMU 506-N. In multiple embodiments, each of the number of multiple output vectors and the corresponding number of multiple input vectors for performing matrix multiplication may be a 3×1 vector. The determined output vector 534 for color correction may also be a 3×1 vector.

[0084] Figure 6 is a block diagram illustrating a configuration example for performing a plurality of color correction operations 670 on a single MMU 606 of an image processor 604-2 formed in a memory cell array according to various embodiments of the present disclosure. Figure 6 The embodiment of the image processor 604-2 described in Figure 2 The crossbar array 211 is described as being formed in association to include a single MMU 606. The image processor 604-2 may be configured to be coupled (eg, directly) to a digital image sensor (eg, sensor circuit 102) to receive a plurality of bits 603 (eg, of an input vector 631).

[0085] The image processor 604-2 may be formed to include a parameter matrix component 672 configured to store a plurality of optional parameter matrices (eg, as shown at 333 and in conjunction with Figure 3 Describe and combine formulas 1-3 and Figure 4 ). The image processor 604-2 may be further formed to include a control unit 671 configured to select from a plurality of selectable parameter matrices 672 and direct input 678 to a single MMU 606 of the selected parameter matrix.

[0086] In various embodiments, image processor 604-2 may be configured to couple to multiple sensors to receive data values ​​603 / 631 to indicate each level of multiple color components and to perform a sequence of multiple color correction operations on a single MMU 606 (as shown at 440 and in conjunction with Figure 4604-2 may perform a sequence of operations to include performing matrix multiplication on an input vector input from a control unit 671 to a single MMU 606 from a parameter matrix component 672 and a selected parameter matrix to determine a color corrected output vector 673. The adjustment vector (e.g., as described in conjunction with the fifth color correction operation 450 and Figure 4 The gamma parameter described above) may be input by the control unit 671 to the single MMU 606 (eg, as another 3x1 parameter to be used in a matrix multiplication of the 3x1 input vector and the selected 3x3 parameter matrix).

[0087] The single MMU 606 of the image processor 604-2 may (for example, in combination with Figure 2 674 (shown in Figure 675 ) is formed as part of the memory cell array and may be coupled (e.g., directly) to a plurality of sensors. Parameter matrix component 672 and / or control unit 671 of image processor 604-2 may each be directly coupled to a single MMU 606 (e.g., as shown at 678 for parameter matrix component 672 and at 673 and 675 for control unit 671). Control unit 671 may be directly coupled 677 to parameter matrix component 672 so that color correction processing is performed on data values ​​stored on the memory cells of image processor 604-2, and in various embodiments, data values ​​associated with the in-memory color correction processing may not be moved to a host component (not shown) indirectly coupled to the array for processing.

[0088] Each of the plurality of selectable parameter matrices stored by parameter matrix component 672 and selectable by control unit 671 for input to single MMU 606 may contain parameter entries configured to implement a particular color correction operation of the execution sequence. Control unit 671 may be configured to direct single MMU 606 to output an output vector 673 corresponding to a first pixel, the output vector determined by executing a first color correction operation in the sequence for storage by control unit 671. Control unit 671 may be configured to direct input 678 from parameter matrix component 672 to single MMU 606 corresponding to a second selected parameter matrix for a second color correction operation. Control unit 671 may be configured to direct input 675 corresponding to the stored output vector of the first pixel as an input vector for performing the second color correction operation. Thus, a sequence of multiple color correction operations may be performed on single MMU 606 until an expected (e.g., desired) color corrected output vector for the first pixel is determined. Thus, color correction operations may be performed on other pixels by image processor 604-2.

[0089] Figure 77 is a flow chart of an example of a method 780 of an image processor formed in a memory cell array that performs color correction according to an illustration of multiple embodiments of the present disclosure. Unless explicitly stated, the elements of the methods described herein are not constrained to a particular order or sequence. In addition, multiple method embodiments described herein or elements thereof may be performed at the same point in time or at substantially the same point in time.

[0090] At block 782, in various embodiments, method 780 may include transmitting an input vector to an image processor formed in a memory cell array as a plurality of bits corresponding to a plurality of color components of an image pixel (e.g., as combined with Figure 1 At block 784, in various embodiments, method 780 may include performing, by the image processor, a matrix multiplication on the input vector and the parameter matrix. At block 786, in various embodiments, method 780 may include determining, by the image processor, an output vector including a plurality of possibly color-corrected components (e.g., color components that may be corrected by performing a color correction operation, such as in conjunction with Figure 3 and 4 and described elsewhere in this document).

[0091] In various embodiments, method 780 may further include forming the image processor in an array to include an MMU formed as a 3×3 array of ReRAM cells in a crossbar configuration. Method 780 may further include forming a 3×3 crossbar configuration to correspond to a 3×3 configuration of parameter entries in the parameter matrix (e.g., as combined with Figure 3 and 4 As described, alternative numbers of memory cells may be formed as an MMU in a crossbar configuration (e.g., based at least in part on a variable number of bits of an input vector, a variable number of color components, a variable number of bits per pixel, and / or a variable number of parameter entries in a parameter matrix).

[0092] The method 780 may further include executing a sequence of a plurality of color correction operations on the image processor (e.g., as combined with Figure 4 and elsewhere herein). In various embodiments, the sequence may include performing color correction operations to include (e.g., sequentially): a defect correction operation for a pixel having a color value that is significantly different from neighboring pixels; a color interpolation operation to interpolate a plurality of missing color values ​​from neighboring pixels for the pixel; a white balance operation to shift the color values ​​of a plurality of color components of the pixel toward white; and / or a color adjustment operation to shift the output of the color values ​​from the sensor circuit to correspond to the perception of the image pixel by human vision (e.g., as combined with Figure 4 describe).

[0093] In various embodiments, method 780 may further include outputting a color corrected output vector or multiple such output vectors from the image processor for: storage by a memory unit other than a memory unit included in the image processor in the array; further processing (e.g., video compression and other such applications) to form a series of images as a video presentation; further processing to form a single image as a still presentation; further processing in an image recognition operation; and / or further processing by a host processor indirectly coupled to the array including the image processor.

[0094] Method 780 may include, in addition to outputting the color-corrected output vector, performing a sequence of a plurality of color correction operations on the image processor to further include, in various embodiments, at least one of: a gamma / brightness / contrast adjustment operation for shifting the brightness and contrast of a plurality of pixels in the image by adjusting a gamma parameter; a color conversion operation for converting the number of a plurality of color components of an image pixel provided by the sensor circuit to a corresponding number of a plurality of color components in a different color space that is more suitable for further processing; and / or a downsampling operation for reducing the color value of the image pixel to a lower number of color values ​​(e.g., as combined with the color value of the image pixel in the different color space) for at least one of the plurality of color components in the different color space. Figure 4 and described elsewhere herein).

[0095] In the above detailed description of the present disclosure, reference is made to the accompanying drawings which form a part of the present disclosure and in which are shown by way of illustration the manner in which one or more embodiments of the present disclosure may be practiced. These embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the embodiments of the present disclosure, and it is understood that other embodiments may be utilized and process, electrical and / or structural changes may be made without departing from the scope of the present disclosure.

[0096] As used herein, particularly with respect to the accompanying drawings, reference numerals with hyphens and / or designators, such as "M," "N," "X," "Y," etc. (e.g., Figure 5 506-1, 506-2, ..., 506-N) in the indication may include multiple specific features so specified. In addition, in some embodiments, when only the first three digits (e.g., 506) are used without hyphens, these digits are presented to generally represent all multiple specific features.

[0097] It should also be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used herein, unless the context clearly dictates otherwise, the singular forms "a", "an" and "the" include single and plural referents, such as "plurality", "at least one" and "one or more" (e.g., a plurality of memory arrays may refer to one or more memory arrays), and "plurality" is intended to refer to more than one such thing. In addition, throughout this application, the word "may" is used in a permissive sense, i.e., with the potential to be able to, rather than in a mandatory sense, i.e., must. The term "including" and its derivatives mean "including but not limited to". The term "coupled / coupling" means physically connected directly or indirectly for access and / or for moving (transmitting) instructions (e.g., control signals, address signals, etc.) and data when the context is appropriate. The terms "data" and "data value" are used interchangeably herein and may have the same meaning (e.g., one or more data units or "bits") when the context is appropriate.

[0098] Although example embodiments suitable for use in a color correction processor have been illustrated and described herein, including combinations and configurations of memory resources, processing resources, a color correction processor (CPU), a matrix multiplication unit (MMU), a ReRAM unit, a crossbar array, a parameter matrix, an input vector, an output vector, a controller, a control unit, and parameter matrix components, as well as other components, embodiments of the present disclosure are not limited to those combinations explicitly recited herein. Other combinations and configurations of memory resources, processing resources, CPUs, MMUs, ReRAM units, crossbar arrays, parameter matrices, input vectors, output vectors, controllers, control units, and parameter matrix components suitable for use in the color correction processor disclosed herein are precisely included within the scope of the present disclosure.

[0099] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that arrangements calculated to achieve the same results may replace the specific embodiments shown. The present disclosure is intended to cover modifications or variations of one or more embodiments of the present disclosure. It should be understood that the above description is provided in an illustrative and not restrictive manner. It will be apparent to those skilled in the art that the combination of the above embodiments and other embodiments not specifically described herein will be apparent after reviewing the above description. The scope of one or more embodiments of the present disclosure includes other applications using the above structures and processes. Therefore, the scope of one or more embodiments of the present disclosure should be determined with reference to the attached claims together with the full scope of equivalents given by such claims.

[0100] In the foregoing detailed description, some features are grouped together in a single embodiment for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the disclosed embodiments of the disclosure must use more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the subject matter of the present invention lies in less than all features of a single disclosed embodiment. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.

Claims

1. An image processing device, comprising: a sensor circuit configured to provide an input vector as a plurality of bits corresponding to a plurality of color components of an image pixel; an image processor formed in an array of memory cells, wherein a number of the memory cells are formed as a matrix multiplication unit MMU, and the number of the memory cells corresponds to the number of parameter entries in the parameter matrix; and The image processor: coupled to the sensor circuit to receive the plurality of bits of the input vector; and Configured to perform a color correction operation in the array by performing a matrix multiplication on the input vector and the parameter matrix to determine a color-corrected output vector.

2. The image processing device according to claim 1, wherein: the number of parameter entries in the parameter matrix corresponding to the number of the plurality of bits in the input vector; The specific parameter entries of the parameter matrix are stored by corresponding memory units in the MMU; the number of the memory cells in the MMU being based at least in part on the number of the plurality of bits in the input vector; and The memory cells in the MMU are arranged in a straight line configuration, wherein an edge of each of two directions of the MMU is formed to include the number of the memory cells corresponding to the number of the plurality of bits.

3. The image processing device according to claim 1, wherein: The plurality of color components of the image pixels correspond to three color components; The plurality of bits of the input vector are three bits to correspond to the three color components; and The image processor is configured to perform a matrix multiplication on the input vector and the parameter matrix to generate the output vector to include three color corrected bits.

4. The image processing apparatus according to claim 1, further comprising: A controller (107) configured to direct: Input of the input vector from the sensor circuit arrives at the image processor; and A plurality of color correction operations are performed in the array by the image processor to determine the color corrected output vector.

5. The image processing apparatus according to claim 1, wherein: A memory unit included in the image processor is configured to perform determining the color-corrected output vector; and The determination of the color-corrected output vector is performed in the array without the memory cells being coupled to a sensing circuit including a sense amplifier.

6. An image processing system, comprising: a light sensor arranged in a color filter array for digital acquisition of a color image, wherein the light sensor is configured to provide a plurality of bits corresponding to a plurality of color components of an image pixel; An image processor formed in an array of memory cells as a plurality of matrix multiplication units MMU coupled in series, wherein each MMU of the plurality of serially coupled MMUs comprises a number of said memory cells, and the number of said memory cells corresponds to the number of parameter entries in the parameter matrix; and The image processor is configured to: coupled to the color filter array to receive the plurality of bits; performing one of a sequence of a plurality of color correction operations on each of the plurality of serially coupled MMUs; and A matrix multiplication is performed on an input vector and the parameter matrix at each of the plurality of serially coupled MMUs to determine a color-corrected output vector.

7. The image processing system according to claim 6, wherein: Each of the plurality of input vectors corresponds to an output vector determined by performing a previous color correction operation on the MMU; and The matrix multiplication is performed on each of the plurality of input vectors and a particular parameter matrix stored by the MMU to achieve performing a particular one of the plurality of color correction operations until a final color-corrected output vector of pixels of the image is determined.

8. The image processing system of claim 6, wherein the sequence of the plurality of color correction operations is performed without feedback from subsequent color correction operations performed in the sequence.

9. The image processing system of claim 6, wherein the memory cell array including the image processor is formed as a resistive random access memory (ReRAM) having memory cells configured to store data values ​​and perform the color correction operation.

10. The image processing system according to claim 6, wherein: The plurality of the serially coupled MMUs are formed in a pipeline configuration (560) wherein each MMU except the last MMU is coupled to the next MMU; A first MMU (560-1) of the pipeline is configured to perform a first color correction operation in the sequence on a first input vector; a second MMU (560-2) of the pipeline configured to perform a second color correction operation in the sequence on a first output vector received from the first MMU as a second input vector of the second MMU; and The first MMU is configured to repeatedly perform the first color correction operation in the sequence on a third input vector in the same clock cycle with the performing the second color correction operation on the second MMU.

11. The image processing system according to claim 6, wherein: The last output vector provided by the last MMU of the pipeline is outputted from the last MMU of the pipeline as the color-corrected output vector, such that: inputting a first input vector to a first MMU of the pipeline, and continuously moving input and output vectors through the sequence of the plurality of color correction operations; and The color corrected output vector is output from the plurality of serially coupled MMUs for storage or further processing.

12. The image processing system according to claim 6, wherein: Each of the number of output vectors and the corresponding number of input vectors for performing the matrix multiplication is a 3x1 vector (332); and The determined color-corrected output vector is also a 3x1 vector (336).

13. An image processing system, comprising: digital image sensors for digitally capturing a color image, wherein each of the digital image sensors is configured to provide a data value indicative of a level of one of a plurality of color components of an image pixel; and An image processor formed in association with an array of memory cells to include: Single matrix multiplication unit MMU; a parameter matrix component configured to store a plurality of selectable parameter matrices; a control unit configured to select from the plurality of selectable parameter matrices and to direct input to the single MMU of the selected parameter matrix; and The image processor is configured to: coupled to a plurality of said digital image sensors to receive data values ​​indicative of each level of said plurality of color components; performing a sequence of a plurality of color correction operations on the single MMU; and Matrix multiplication is performed on an input vector input by the control unit from the parameter matrix components to the single MMU and the selected parameter matrix to determine a color corrected output vector.

14. The image processing system according to claim 13, wherein: The single MMU is formed in the memory cell array as a 3×3 array of resistive random access memory ReRAM cells in a crossbar configuration; and Each of the plurality of selectable parameter matrices stored by the parameter matrix component and selectable by the control unit for input to the single MMU includes nine (3x3) parameter entries to be stored on the 3x3 array.

15. The image processing system of claim 13, wherein a 3x1 adjustment vector is output by the control unit to the single MMU as another parameter to be used for the matrix multiplication of the 3x1 input vector and the selected 3x3 parameter matrix.

16. The image processing system according to claim 13, wherein: The single MMU of the image processor is formed as part of the memory cell array and is directly coupled to the plurality of the digital image sensors; and The parameter matrix components of the image processor and the control unit are each directly coupled to the single MMU, and the control unit is directly coupled to the parameter matrix components so that data values ​​related to color correction processing are not moved to a host component indirectly coupled to the array for processing.

17. The image processing system of claim 13, wherein each of the plurality of selectable parameter matrices stored by the parameter matrix component and selectable by the control unit for input to the single MMU comprises parameter entries configured to enable execution of a particular color correction operation in the sequence.

18. The image processing system according to claim 13, wherein: the control unit being configured to direct the single MMU to output an output vector corresponding to a first pixel, the output vector being determined by performing a first color correction operation in the sequence for storage by the control unit; the control unit being configured to direct input from the parameter matrix components to the single MMU corresponding to a second selected parameter matrix for a second color correction operation; the control unit being configured to direct input of a stored output vector corresponding to the first pixel as an input vector for performing the second color correction operation; and Thus, the sequence of the plurality of color correction operations is performed on the single MMU until the color-corrected output vector for the first pixel is determined.

19. The image processing system according to claim 13, wherein: The memory cells included in the crossbar switch configuration of the resistive random access memory (ReRAM) cells of the single MMU include: a top electrode coupled to the source component to apply a specific voltage potential; a dielectric metal oxide, forming a conductive path through the dielectric metal oxide by applying the specific voltage potential; and A bottom electrode is coupled to a ground component to act as a drain.

20. A method of operating a memory device for image processing, comprising: forming an image processor in an array of memory cells to include a matrix multiplication unit (MMU) formed as an array of resistive random access memory (ReRAM) cells in a crossbar configuration; forming the crossbar configuration to correspond to the configuration of parameter entries in the parameter matrix; transmitting an input vector to the image processor as a plurality of bits corresponding to a plurality of color components of an image pixel; performing, by the image processor, matrix multiplication on the input vector and the parameter matrix; and An output vector comprising a plurality of color corrected components is determined by the image processor based at least in part on the matrix multiplication.

21. The method of claim 20, wherein: The array is formed as a 3×3 array of Resistive Random Access Memory (ReRAM) cells in the crossbar configuration; and The crossbar configuration corresponds to a 3x3 crossbar configuration, which corresponds to a 3x3 configuration of parameter entries in the parameter matrix.

22. The method of claim 20, further comprising: performing a sequence of a plurality of color correction operations on the image processor, wherein the sequence comprises performing: Defect correction operations performed on pixels having color values ​​that differ significantly from neighboring pixels; a color interpolation operation for interpolating, for the pixel, a plurality of missing color values ​​from neighboring pixels; a white balance operation for shifting the color values ​​of the plurality of color components of the pixel toward white; and A color adjustment operation is used to shift the output of the color value from the sensor circuit to correspond to the perception of the image pixel by human vision.

23. The method of claim 22, further comprising outputting a color corrected output vector from the image processor for storage by a memory cell in the array other than a memory cell included in the image processor.

24. The method of claim 22, further comprising outputting a color corrected output vector from the image processor for further processing by a host processor indirectly coupled to the array including the image processor.

25. The method of claim 22, further comprising: Wherein executing the sequence further comprises executing: Gamma / brightness / contrast adjustment operation, used to shift the brightness and contrast of multiple pixels in an image by adjusting the gamma parameter; a color conversion operation for converting the number of the plurality of color components of the image pixel provided by the sensor circuit to a corresponding number of a plurality of color components in a different color space more suitable for further processing; and A downsampling operation is used to reduce the color values ​​comprising the image pixels to a lower number of color values ​​for at least one of the plurality of color components in the different color space.

Images