Demosaic operation circuit, image sensing device, and operation method thereof

Abstract

To provide a circuit which performs a demosaic operation to an image pattern in which a white pixel is 50%.SOLUTION: A circuit 300 includes: a white pixel value estimation unit 310 which acquires an RGB channel average value and a gradient value of a pixel for each direction using source pixel data provided from a pixel array, and estimating a white pixel value based on the RGB channel average value; a fine adjustment unit 320 which finely adjusts an estimated white pixel value by removing noise through a different filter based on weighting which is differently allocated according to a gradient of a center pixel and a neighboring white pixel; a chroma estimation unit 330 which estimates a chroma for each channel by calculating a chroma array based on the source pixel data and the adjusted white pixel value; and a color phase correction unit 340 which corrects a color phase based on the white pixel value estimated by the fine adjustment unit 320 and the chroma weighting estimated by the chroma estimation unit 330.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a semiconductor device, and more particularly to a demosaic operating circuit, an image sensing device, and an operating method thereof. [Background technology]

[0002] In recent years, the paradigm for computing environments has shifted to ubiquitous computing, which allows computer systems to be used anywhere, anytime, and this has led to a rapid increase in the use of portable electronic devices such as mobile phones, digital cameras, and notebook computers.

[0003] In particular, the rapid development of video equipment has accelerated the development of image capturing devices such as cameras and camcorders equipped with image sensors. These image capturing devices are capable of capturing images, recording them on a recording medium, and playing them back at any time, and the number of users of these devices is rapidly increasing. As a result, users' demands for performance and functionality are gradually increasing, and they are pursuing smaller, lighter, and lower power consumption devices as well as higher performance and multiple functions. Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments of the present invention may provide a demosaic operation circuit, an image sensing device, and an operation method thereof that perform demosaic operation on an image pattern in which white pixels account for 50% of source pixel data having an RGBW pattern.

[0005] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0006] The present invention provides a demosaicing circuit, an image sensing device, and a method of operating the same.

[0007] A demosaic operation circuit according to embodiments of the present invention may include a white pixel value estimation unit that acquires a pixel gradient value and an RGB channel average value for each direction using source pixel data provided from a pixel array including a plurality of pixels and estimates a white pixel value based on the acquired gradient value and an RGB channel average value; a fine adjustment unit that removes noise through different filters based on weights that are differently assigned depending on the gradients of a center pixel and surrounding white pixels and fine-adjusts the estimated white pixel value; a chroma estimation unit that calculates a chroma array based on the source pixel data and the adjusted white pixel value and estimates chroma for each channel; and a hue correction unit that corrects hue based on the white pixel value estimated by the fine adjustment unit and the chroma weights estimated by the chroma estimation unit.

[0008] The white pixel value estimating unit may include a direction-specific white pixel value calculation block that calculates a gradient value of a white pixel for each direction based on a difference value between pixels having different hues and an absolute value of a difference value between white pixels among source pixel data provided from the pixel array; an RGB channel average value calculation block that calculates an RGB channel average value according to a position of a center pixel among source pixel data provided from the pixel array; and a white pixel value estimation block that estimates a white pixel value based on the gradient value of a white pixel for each direction calculated by the direction-specific white pixel value calculation block and the RGB channel average value calculated by the RGB channel average value calculation block.

[0009] The white pixel value estimation block can estimate a white pixel value corresponding to the center pixel using a horizontal filter and a vertical filter when the center pixel is a green pixel, a red pixel, or a blue pixel.

[0010] The fine adjustment unit may finely adjust a white pixel value using a gradient value of the white pixel estimated by the white pixel value estimation unit when the center pixel is a green pixel, a red pixel, or a blue pixel.

[0011] The chroma estimator may calculate a chroma array by subtracting the refined white pixel value from the source pixel data.

[0012] The chroma estimator may assign chroma weights according to chroma similarities for each channel, centered on a chroma value of a center pixel in the chroma array, and estimate chromas for each channel according to the weights.

[0013] According to another embodiment of the present invention, an image sensing apparatus includes an image sensor having a plurality of pixels and an image signal processor that processes an output signal of the image sensor. A demosaic operation circuit is implemented within one of the image sensor and the image signal processor. The demosaic operation circuit may include: a white pixel value estimation unit that acquires a gradient value of a white pixel for each direction and an average value of RGB channels using source pixel data provided from a pixel array having a plurality of pixels and estimates a white pixel value based on the gradient value and an average value of RGB channels; a fine adjustment unit that removes noise through different filters based on weights that are assigned differently depending on gradients of a center pixel and peripheral white pixels and fine-adjusts the estimated white pixel value; a chroma estimation unit that calculates a chroma array based on the source pixel data and the fine-adjusted white pixel value and estimates chroma for each channel; and a hue correction unit that corrects hue based on the white pixel value estimated by the fine adjustment unit and the chroma weighting estimated by the chroma estimation unit.

[0014] The white pixel value estimating unit may include a direction-specific white pixel value calculation block that calculates a gradient value of a white pixel for each direction based on an absolute value of a difference value between pixels having different hues and a difference value between white pixels among source pixel data provided from the pixel array; an RGB channel average value calculation block that calculates an RGB channel average value according to a position of a center pixel among source pixel data provided from the pixel array; and a white pixel value estimation block that estimates a white pixel value based on the gradient value of the white pixel for each direction calculated by the direction-specific white pixel value calculation block and the RGB channel average value calculated by the RGB channel average value calculation block.

[0015] The white pixel value estimation block can estimate a white pixel value corresponding to the center pixel using a horizontal filter and a vertical filter when the center pixel is a green pixel, a red pixel, or a blue pixel.

[0016] The fine adjustment unit may finely adjust a white pixel value using the white pixel value estimated by the white pixel value estimation unit when the center pixel is a green pixel, a red pixel, or a blue pixel.

[0017] The chroma estimator may calculate a chroma array by subtracting the adjusted white pixel value from the source pixel data.

[0018] The chroma estimator may assign chroma weights according to chroma similarities for each channel, centered on a chroma value of a center pixel in the chroma array, and estimate chromas for each channel according to the weights.

[0019] According to another embodiment of the present invention, a method for operating an image sensing device may include the steps of: acquiring a gradient value of a white pixel for each direction and an average value of RGB channels using source pixel data provided from a pixel array, and estimating a white pixel value based on the gradient value; removing noise using different filters based on weights assigned differently depending on gradients of a center pixel and peripheral white pixels of the pixel array, and fine-tuning the estimated white pixel value; calculating a chroma array based on the source pixel data and the fine-tuned white pixel value, and estimating chroma for each channel; and correcting a hue based on the white pixel value estimated by the fine-tuning unit and the chroma weights estimated by the chroma estimation unit.

[0020] The step of estimating the white pixel value includes the steps of: calculating a gradient value of a white pixel for each direction based on a difference value between pixels having different hues and a difference value between white pixels among source pixel data provided from the pixel array; and calculating an RGB channel average value according to a position of a center pixel among source pixel data provided from the pixel array. The white pixel value can be estimated based on the directional white pixel value calculated in the step of calculating the directional white pixel value and the RGB channel average value calculated in the step of calculating the RGB channel average value.

[0021] In the step of estimating the white pixel value, if the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value corresponding to the center pixel can be estimated using a horizontal filter and a vertical filter.

[0022] In the step of fine-adjusting the white pixel value, if the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value can be fine-adjusted using the white pixel value estimated by the white pixel value estimation unit.

[0023] The step of estimating the per-channel chroma may calculate a chroma array by subtracting the adjusted white pixel value from the source pixel data.

[0024] In the step of estimating the chroma for each channel, a chroma weight may be assigned according to the chroma similarity for each channel, centered on the chroma value of a center pixel in the chroma array, and the chroma for each channel may be estimated according to the weight.

[0025] The effects that can be obtained in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Effects of the Invention]

[0026] The embodiments of the present invention can perform demosaicing on various RGBW patterns with 50% white pixels, thereby improving the applicability of image sensing devices. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 shows a block diagram of a demosaic operation circuit according to an embodiment of the present invention. [Figure 2] 10A and 10B are diagrams illustrating calculation of white pixels by direction based on difference values ​​between pixels having different hues; [Figure 3] 10A and 10B are diagrams illustrating calculation of white pixels by direction based on differences between white pixels; [Figure 4] FIG. 10 shows an estimated white pixel for a centrally located red pixel. [Figure 5] FIG. 10 is a diagram illustrating fine adjustment of a white pixel. [Figure 6] FIG. 10 is a diagram illustrating estimation of a chroma array. [Figure 7] 10 is a graph showing white pixel gain, edge gain, and hue gain. [Figure 8] FIG. 10 shows a block diagram of an image sensing device according to another embodiment of the present invention. [Figure 9] FIG. 10 shows a block diagram of an image sensing device according to yet another embodiment of the present invention. [Figure 10] 10 is a flowchart illustrating an operation of an image sensing apparatus according to yet another embodiment of the present invention. [Figure 11] 1 shows a block diagram illustrating an embodiment of a system to which an image sensing device according to an embodiment of the present invention is applied; DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that in the following description, only the parts necessary for understanding the operation of the present invention will be described, and the description of the other parts will be omitted so as not to obscure the gist of the present invention.

[0029] Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.

[0030] A demosaic operation circuit according to one embodiment of the present invention will be described with reference to FIGS.

[0031] FIG. 1 shows a block diagram of a demosaic operation circuit according to an embodiment of the present invention, FIG. 2 is a diagram illustrating estimation of a white pixel based on the difference between pixels having different hues, FIG. 3 is a diagram illustrating estimation of a white pixel based on the difference between white pixels, FIG. 4 is a diagram illustrating an estimated white pixel of a red pixel located at the center, FIG. 5 is a diagram illustrating fine adjustment of the white pixel, FIG. 6 is a diagram illustrating estimation of a chroma array, and FIG. 7 is a graph showing white pixel gain, edge gain, and hue gain.

[0032] The demosaic operation circuit 300 according to an embodiment of the present invention shown in FIG. 1 may include a white pixel value estimator 310, a fine adjuster 320, a chroma estimator 330, and a hue corrector 340.

[0033] The white pixel value estimation unit 310 can estimate a white pixel value based on a gradient value and an RGB channel average value of a white pixel for each direction using source pixel data provided from a pixel array (not shown) having a plurality of pixels. The white pixel value estimation unit 310 can include a direction white pixel value calculation block 312, an RGB channel average value calculation block 314, and a white pixel value estimation block 316.

[0034] The direction-specific white pixel value calculation block 312 may calculate a gradient value of a direction-specific white pixel based on a difference value between pixels having different hues and a difference value between white pixels among source pixel data provided from the pixel array.

[0035] Referring to FIG. 2, the gradient value of the white pixel for each direction can be calculated using the difference value between pixels having different hues as shown in Equation 1.

number

[0036] In addition, the slope value of the white pixel in the west direction, the slope value of the white pixel in the south direction, and the slope value of the white pixel in the north direction can be calculated in the same manner as in Equation 1.

[0037] Referring to FIG. 3, the gradient value of the white pixels for each direction can be calculated using the difference value between the white pixels as shown in Equation 2 below.

number

[0038] The RGB channel average value calculation block 314 can calculate the average value of each RGB channel according to the position of the center pixel of the source pixel data provided from the pixel array.

[0039] The white pixel value estimation block 316 can estimate a white pixel value based on the gradient value of the white pixel for each direction calculated in the direction-specific white pixel value calculation block 312 and the RGB channel average value calculated in the RGB channel average value calculation block 314 .

[0040] The estimated horizontal white pixel value can be calculated and estimated according to Equation 3 below.

number

[0041] The estimated vertical white pixel value can be calculated and estimated according to Equation 4 below.

number

[0042] The estimated horizontal / vertical white pixel values ​​can be calculated and estimated according to Equation 5 below.

number

[0043] The fine adjustment unit 320 can remove noise using different filters based on weights assigned differently depending on the gradients of the center pixel and the surrounding white pixels, and can fine-adjust the estimated white pixel value.

[0044] For example, as shown in FIG. 4, the estimated white pixel value Wc can be fine-tuned using a red pixel located at the center of a 7×7 kernel.

[0045] In particular, if the center pixel is a red pixel (Red pi×el), blue pixel (Blue pi×el), or green pixel (Green pi×el), the white pixel value estimated by the white pixel value estimation unit 310 can be used to remove noise through a horizontal filter, a vertical filter, or a horizontal / vertical filter, and the estimated white pixel value can be fine-tuned.

[0046] As shown in FIG. 5, if the sum of the slope values ​​of the white pixels in the east direction and the west direction multiplied by a predetermined weighting factor α is smaller than the sum of the slope values ​​of the white pixels in the south direction and the west direction, the noise can be removed through a horizontal filter and the estimated white pixel value can be fine-tuned.

[0047] In addition, if the sum of the slope values ​​of the white pixels in the east direction and the west direction multiplied by a predetermined weighting factor α is greater than the sum of the slope values ​​of the white pixels in the south direction and the west direction, noise can be removed through a vertical filter, and the estimated white pixel value can be fine-tuned.

[0048] In addition, if the sum of the slope values ​​of the white pixels in the east direction and the west direction multiplied by a predetermined weighting factor α is equal to the sum of the slope values ​​of the white pixels in the south direction and the west direction, noise can be removed using horizontal / vertical filters, and the estimated white pixel value can be fine-tuned.

[0049] The chroma estimator 330 calculates a chroma array based on the source pixel data and the adjusted white pixel value, and can estimate the chroma for each channel.

[0050] More specifically, as shown in FIG. 6, the chroma estimation unit 330 calculates a chroma array by subtracting an adjusted white pixel value (White) from source pixel data of a Bayer pattern, assigns chroma weighting based on the chroma similarity for each channel, centered on the chroma value of a center pixel in the chroma array, and estimates chroma for each channel according to the weighting.

[0051] The chroma for each channel can be estimated by Equation 6 below.

number

[0052] The hue correction unit 340 can correct the hue based on the white pixel value estimated by the fine adjustment unit 320 and the chroma weighting estimated by the chroma estimation unit 330 .

[0053] As shown in Figure 7, the white pixel gain (DY gain ), Edge Gain (Edge gain ), Hue Gain (Color gain ) to optimize the hue correction.

[0054] That is, the hue correction can be calculated as shown in Equation 7 below.

number

[0055] FIG. 8 shows a block diagram of an image sensing device according to an embodiment of the present invention.

[0056] As shown in FIG. 8, the image sensing device 10 may include an image sensor 100 and an image signal processor (ISP) 400.

[0057] The image sensing device 10 may be implemented as a personal computer (PC) or a mobile computing device, such as a laptop computer, a mobile phone, a smartphone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, a portable multimedia player (PMP), a mobile internet device (MID), a wearable computer, an internet of things (IoT) device, or an internet of everything (IoE) device.

[0058] The image sensor 100 shown in FIG. 8 may include a pixel array 200 and a demosaic operation circuit 300.

[0059] The pixel array 200 may include a plurality of pixels, where pixel may refer to pixel data, and may have, but is not limited to, an RGB data format, a YUV data format, or a YCbCr data format.

[0060] The demosaic operation circuit 300 performs demosaic operation on an RGBW pattern in which white pixels account for 50%.

[0061] The detailed configuration and operation of the demosaic operating circuit 300 are substantially the same as or similar to the configuration and operation of the demosaic operating circuit 300 shown in FIGS. 1 to 7, and therefore detailed description thereof will be omitted.

[0062] The image signal processor 400 is an embodiment of a processor and can be realized as an integrated circuit, a system on chip (SoC), or a mobile AP. The image signal processor 400 processes the output signal of the image sensor 100. That is, the image signal processor 400 receives and processes an image output signal that has undergone demosaic operation of an RGBW pattern in which white pixels account for 50% in the demosaic operation circuit 300 provided in the image sensor 100.

[0063] In particular, the image signal processor 400 can process a Bayer pattern (BAYER) corresponding to pixel data to generate RGB image data. For example, the image signal processor 400 can process the Bayer pattern (BAYER) so that the image data (IDATA) can be displayed on a display, and transmit the processed image data to an interface.

[0064] In some embodiments, the image sensor 100 and the image signal processor 400 may each be implemented as a chip and may be implemented in a single package, such as a multi-chip package (MCP). In other embodiments, the image sensor 100 and the image signal processor 400 may be implemented in a single chip.

[0065] FIG. 9 shows a block diagram of an image sensing device according to another embodiment of the present invention.

[0066] As shown in FIG. 9, the image sensing device 10 may include an image sensor 100 and an image signal processor (ISP) 400.

[0067] Except for the fact that the demosaic operation circuit 300 is implemented in the image signal processor 400 rather than in the image sensor 100, the structure and operation of the image sensing device 10 of Figure 9 are substantially the same as or similar to the structure and operation of the image sensing device 10 of Figure 8, and therefore detailed description thereof will be omitted.

[0068] Hereinafter, the operation of the image sensing apparatus according to yet another embodiment of the present invention will be described with reference to Fig. 10. Fig. 10 is a flowchart illustrating the operation of the image sensing apparatus according to yet another embodiment of the present invention.

[0069] The operation of the image sensing device shown in FIG. 10 includes a step of estimating a white pixel value (S1000), a step of fine-tuning the estimated white pixel value (S1100), a step of estimating chroma for each channel (S1200), and a step of correcting hue (S1300).

[0070] In the step of estimating a white pixel value (S1000), the gradient value and RGB channel average value of the white pixel for each direction are obtained using source pixel data provided from the pixel array, and the white pixel value can be estimated based on the gradient value and RGB channel average value.

[0071] The step of estimating the white pixel value may include a step of calculating a gradient value of the white pixel for each direction (S1010), a step of calculating an average value of the RGB channels (S1020), and a step of estimating the white pixel value (S1030).

[0072] In the step S1010 of calculating the gradient value of the white pixel for each direction, the gradient value of the white pixel for each direction can be calculated based on the difference value between pixels having different hues and the difference value between white pixels among the source pixel data provided from the pixel array.

[0073] In the step of calculating the RGB channel average values, the RGB channel average values ​​can be calculated according to the position of the center pixel of the source pixel data provided from the pixel array.

[0074] In the white pixel value estimation step, the white pixel value can be estimated based on the directional white pixel gradient value calculated in the directional white pixel value calculation step and the RGB channel average value calculated in the RGB channel average value calculation step.

[0075] In this case, if the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value corresponding to the center pixel can be estimated using a horizontal filter and a vertical filter.

[0076] In the step of fine-tuning the white pixel value (S1100), noise can be removed using different filters based on weights assigned differently depending on the gradients of the center pixel and the surrounding white pixels of the pixel array, and the estimated white pixel value can be fine-tuned.

[0077] At this time, if the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value can be finely adjusted using the white pixel value estimated in the white pixel estimation step.

[0078] In the step of estimating chroma (S1200), a chroma array can be calculated based on the source pixel data and the refined white pixel value to estimate per-channel chroma.

[0079] At this time, the chroma array can be calculated by subtracting the adjusted white pixel value from the source pixel data.

[0080] In addition, chroma weighting is assigned according to the chroma similarity for each channel, centered on the chroma value of the center pixel of the chroma array, and the chroma for each channel can be estimated according to the weighting.

[0081] An embodiment of a system to which an image sensing device according to an embodiment of the present invention is applied will be described below. Fig. 11 shows a block diagram illustrating an embodiment of a system to which an image sensing device according to an embodiment of the present invention is applied.

[0082] The system shown in FIG. 11 can be any of a variety of types of computing devices, including, but not limited to, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computing device, a cellular phone, a smartphone, a mobile phone, a workstation, a network computer, a consumer device, an application server, a storage device, an intelligent display, a peripheral device such as a switch, modem, router, etc., or generally any type of computing device.

[0083] According to one embodiment, the system illustrated in FIG. 11 may represent a system-on-a-chip (SOC). As the name implies, the components of the SOC (1000) may be integrated onto a single semiconductor substrate, such as an integrated circuit "chip." In some embodiments, the components may be implemented on two or more separate chips in the system. The SOC (1000) will be used herein as an example.

[0084] In the illustrated embodiment, the components of the SOC (1000) may include a central processing unit (CPU) complex 1020, on-chip peripheral components 1040A and 1040B (more simply, "peripherals"), a memory controller (MC) 1030, an image signal processor 400, and a communications fabric 1010.

[0085] The SOC (1000) may be further coupled to additional components such as memory 1800 and image sensor 100. All of the components (1020, 1030, 1040A and 1040B, and 200) may be coupled to a communications fabric 1010. The memory controller 1030 may be coupled to memory 1800 during use, and the peripheral device 1040B may be coupled to an external interface 1900 during use.

[0086] In the illustrated embodiment, CPU complex 1020 may include one or more processors 1024 and a level 2 (L2) cache 1022. Peripherals 1040A and 1040B may be any set of additional hardware functionality included in SOC 1000. For example, peripherals 1040A and 1040B may include a display controller configured to display video data on one or more display devices, a graphics processing unit (GPU), a video encoder / decoder, a scaler, a rotator, a blender, etc.

[0087] The image signal processor 400 can process image capture data from the image sensor 100 (or other image sensors). The image signal processor 400 and the image sensor 100 may have the same configurations and operations as the image signal processor 400 and the image sensor 100 shown in FIGS. 1 to 10.

[0088] The peripherals may further include audio peripherals, such as microphones, speakers, interfaces to microphones and speakers, audio processors, digital signal processors, mixers, etc. The peripherals may include a peripheral interface controller (e.g., peripheral 1040B) for various interfaces 1900 external to the SOC (1000), including interfaces such as Universal Serial Bus (USB), Peripheral Component Interconnect (PCI) including PCI Express (PCIe), serial and parallel ports, etc. The peripherals may further include networking peripherals, such as a Media Access Controller (MAC). In general, any set of hardware may be included according to various embodiments.

[0089] The CPU complex 1020 may include one or more CPU processors 1024 that act as the CPU for the SOC (1000). The CPU of a system may include a processor (or processors) that executes the main control software of the system, e.g., an operating system. Generally, during use, the software executed by the CPU may control other components of the system to achieve the intended system functionality. The processor 1024 may also execute other software, e.g., application programs. The application programs may provide user functionality and may rely on the operating system for low-level device control. Thus, the processor 1024 may also be referred to as an application processor.

[0090] CPU complex 1020 may further include interfaces to other hardware, such as an L2 cache 1022 and / or other components of the system (eg, an interface to communications fabric 1010).

[0091] Generally, a processor may include any circuitry and / or microcode configured to execute instructions defined in an instruction set architecture implemented by the processor. Instructions and data operated on by the processor in response to executing instructions may generally be stored in memory 1800, although certain instructions may also be defined for direct processor access to peripheral devices, etc. A processor may encompass a processor core implemented on an integrated circuit with other components, etc., as an integrated part, such as a system-on-chip (SOC) (1000) or other level. A processor may further encompass another microprocessor, processor core, and / or a microprocessor integrated within a multi-chip module implementation, a processor implemented as multiple integrated circuits, etc.

[0092] The memory controller 1030 may generally include circuitry for receiving memory operations from other components of the SOC (1000) and for accessing and performing memory operations on the memory 1800. The memory controller 1030 may be configured to access any type of memory 1800. For example, the memory 1800 may be static random access memory (SRAM), dynamic RAM (DRAM), or synchronous DRAM (SDRAM), including double data rate (DDR, DDR2, DDR3, etc.) DRAM. Low-power / mobile versions of DDR DRAM (e.g., LPDDR, mDDR, etc.) may also be supported. The memory controller 1030 may include queues for memory operations, directing (and potentially redirecting) operations and submitting operations to the memory 1800. The memory controller 1030 may further include data buffers for storing write data awaiting writing to memory and read data awaiting return to the source of the memory operation.

[0093] In some embodiments, the memory controller 1030 may include a memory cache that stores recently accessed memory data. In a SOC implementation, for example, the memory cache may reduce power costs in the SOC by avoiding re-accessing data from memory 1800 if it is expected to be accessed again soon. In some cases, the memory cache may also be referred to as a system cache, as opposed to a private cache that only serves certain components, such as the L2 cache 1022 or cache of the processor 1024. Additionally, in some embodiments, the system cache need not be located within the memory controller 1030.

[0094] In embodiments, memory 1800 can be packaged with SOC (1000) in a chip-on-chip or package-on-package configuration. A multi-chip module configuration of SOC (1000) and memory 1800 can also be used. Such a configuration can be relatively more stable (in terms of data observation) than transmission to other components in the system (e.g., to endpoints 16A and 16B). Thus, protected data can reside unencrypted in memory 1800, while the protected data can be encrypted for exchange between SOC (1000) and external endpoints.

[0095] The communications fabric 1010 can be any communications interconnect and protocol for communication among the components of the SOC (1000). The communications fabric 1010 can be bus-based, including shared bus configurations, crossbar configurations, and hierarchical buses with bridges. The communications fabric 1010 can also be packet-based, hierarchical with bridges, crossbar, point-to-point, or other interconnects. There may be more or fewer individual components / subcomponents than those shown in FIG. 9.

[0096] In some embodiments, the methods described herein can be implemented by a computer program product, or software. In some embodiments, a non-transitory, computer-readable storage medium can store instructions that can be used to program a computer system (or other electronic device) to perform some or all of the techniques described herein. A computer-readable storage medium can include any mechanism for storing information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). Machine-readable media can include, but are not limited to, magnetic storage media (e.g., floppy diskettes), optical storage media (e.g., CD-ROMs), magneto-optical storage media, read-only memory (ROM), random access memory (RAM), erasable and programmable memory (e.g., EPROMs and EEPROMs), flash memory, and other types of media suitable for storing electrical or program instructions. Additionally, program instructions can be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.).

[0097] The computer system 1000 may include a processor unit 1020 (possibly including multiple processors, single-threaded processors, multi-threaded processors, multi-core processors, etc.) that may be configured to execute one or more modules, e.g., noise reduction circuits, that may reside within program instructions stored in memory 1800 of the same computer system, or that may reside within program instructions stored in the memory of yet another computer system similar to or different from the computer system 1000.

[0098] While specific embodiments have been described in the detailed description of the present invention, it is of course possible to make various modifications without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the following claims as well as equivalents to the claims. [Explanation of symbols]

[0099] 10 Image sensing device 100 image sensors 200 pixel array 300 Demosaic Operation Circuit 310 White pixel value estimation unit 320 Fine adjustment section 330 Chroma Estimation Unit 340 Hue correction section 400 Image Signal Processor

Claims

1. a white pixel value estimation unit that acquires a gradient value of a white pixel for each direction using source pixel data provided from a pixel array having a plurality of pixels, acquires an average value of RGB channels according to a position of a center pixel from the source pixel data, and estimates a white pixel value corresponding to the center pixel based on the average value of RGB channels; a fine adjustment unit for removing noise from the white pixels through different filters based on weights assigned differently depending on the gradients of the center pixel and the surrounding white pixels, and fine-adjusting the white pixel values; a chroma estimation unit that calculates a chroma array based on the source pixel data and the adjusted white pixel value, assigns chroma weights to each channel, and estimates chroma according to the weights; a hue correction unit that corrects the hue based on the white pixel value adjusted by the fine adjustment unit and the chroma estimated by the chroma estimation unit; Equipped with The demosaic operation circuit, wherein the RGB channel average values ​​represent average values ​​of pixel values ​​corresponding to each of the red, green, and blue color channels.

2. The white pixel value estimation unit a direction-specific white pixel value calculation block that calculates a gradient value of the white pixel for each direction based on absolute values ​​of difference values ​​between pixels having different hues and difference values ​​between white pixels among source pixel data provided from the pixel array; an RGB channel average value calculation block that calculates the RGB channel average values ​​according to the position of the center pixel among source pixel data provided from the pixel array; a white pixel value estimation block that estimates a white pixel value based on the directional white pixel gradient value calculated by the directional white pixel value calculation block and the RGB channel average value calculated by the RGB channel average value calculation block; The demosaic operation circuit according to claim 1 , comprising:

3. The white pixel value estimation block:

3. The demosaic operation circuit according to claim 2, wherein when the center pixel is a green pixel, a red pixel, or a blue pixel, a white pixel value corresponding to the center pixel is estimated using a horizontal filter and a vertical filter.

4. The fine adjustment unit is 4. The demosaic operation circuit according to claim 3, wherein when the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value is finely adjusted using the gradient value of the white pixel estimated by the white pixel value estimation unit.

5. The chroma estimator 2. The demosaicing circuit of claim 1, wherein the chroma array is calculated by subtracting the fine-tuned white pixel value from the source pixel data.

6. The chroma estimator assigning chroma weights according to chroma similarity for each channel centered on the chroma value of a center pixel in the chroma array, and estimating chroma for each channel according to the weights; The demosaic operation circuit according to claim 5 , wherein the chroma similarity indicates a degree of similarity between the chroma of a center pixel and the chroma of a surrounding pixel.

7. The demosaic circuit of claim 6 , wherein the channel-specific chroma is estimated by the following equation: [Equation 1] At this time, R ｃ ｃｈｒｏｍａ represents the estimated chroma of the centered red pixel, and G ｃ ｃｈｒｏｍａ represents the estimated chroma of the centered green pixel, and B ｃ ｃｈｒｏｍａ represents the estimated chroma of the centrally located blue pixel, wgt represents the weighting, and (i, j) represents the positions of the red, green, and blue pixels.

8. The demosaic operation circuit according to claim 7 , wherein the hue correction unit corrects the hue according to the following formula: [Equation 2] At this time, fcs ｇａｉｎ represents the error hue correction gain, and DY ｇａｉｎ represents the white pixel gain, and Edge ｇａｉｎ represents the edge gain, and Color ｇａｉｎ represents the hue gain, and R ｏｕｔ represents the sum of the refined white pixel value and the estimated chroma value of the centered red pixel, and G ｏｕｔ represents the sum of the refined white pixel value and the estimated chroma value of the centered green pixel, and B ｏｕｔ denotes the sum of the refined white pixel value and the estimated chroma value of the centered blue pixel, and R ｅｓｔ represents the corrected pixel value of the red pixel, and B ｅｓｔ represents the corrected pixel value of the blue pixel.

9. an image sensor having a plurality of pixels; an image signal processor for processing an output signal of the image sensor; Equipped with a demosaic operation circuit is implemented within one of the image sensor and the image signal processor; The demosaic operation circuit includes: a white pixel value estimation unit that obtains a gradient value of a white pixel for each direction using source pixel data provided from a pixel array having a plurality of pixels, obtains an average value of RGB channels according to a position of a center pixel from the source pixel data, and estimates a white pixel value corresponding to the center pixel based on the average value of RGB channels; a fine adjustment unit for removing noise from the white pixels through different filters based on weights assigned differently depending on the gradients of the center pixel and the surrounding white pixels, and fine-adjusting the white pixel values; a chroma estimation unit that calculates a chroma array based on the source pixel data and the fine-tuned white pixel value, assigns chroma weights to each channel, and estimates chroma according to the weights; a hue correction unit that corrects the hue based on the white pixel value adjusted by the fine adjustment unit and the chroma estimated by the chroma estimation unit; Equipped with The RGB channel average values ​​represent average values ​​of pixel values ​​corresponding to red, green, and blue color channels of the image sensing device.

10. The white pixel value estimation unit a direction-specific white pixel value calculation block that calculates a gradient value of the white pixel for each direction based on absolute values ​​of difference values ​​between pixels having different hues and difference values ​​between white pixels among source pixel data provided from the pixel array; an RGB channel average value calculation block that calculates the RGB channel average values ​​according to the position of the center pixel among source pixel data provided from the pixel array; a white pixel value estimation block that estimates a white pixel value based on the gradient value of the white pixel for each direction calculated by the white pixel value calculation block for each direction and the RGB channel average value calculated by the RGB channel average value calculation block; The image sensing device of claim 9 , comprising:

11. The white pixel value estimation block:

11. The image sensing device of claim 10, wherein if the center pixel is a green pixel, a red pixel, or a blue pixel, a white pixel value corresponding to the center pixel is estimated using a horizontal filter and a vertical filter.

12. The fine adjustment unit is 12. The image sensing device of claim 11, wherein when the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value is fine-adjusted using the gradient value of the white pixel estimated by the white pixel value estimator.

13. The chroma estimator 10. The image sensing device of claim 9, wherein the chroma array is calculated by subtracting the adjusted white pixel value from the source pixel data.

14. The chroma estimator assigning chroma weights according to chroma similarities for each channel centered on the chroma value of a center pixel in the chroma array, and estimating chromas for each channel according to the weights; The image sensing device of claim 13 , wherein the chroma similarity represents a degree of similarity between the chroma of a center pixel and the chroma of a surrounding pixel.

15. obtaining a gradient value of a white pixel for each direction using source pixel data provided from a pixel array, obtaining RGB channel average values ​​according to the position of a center pixel from the source pixel data, and estimating a white pixel value corresponding to the center pixel based on the average values; removing noise from the white pixels through different filters based on weights assigned differently according to the gradients of the center pixel and the surrounding white pixels of the pixel array, and fine-tuning the white pixel values; calculating a chroma array based on the source pixel data and the refined white pixel value, assigning chroma weights to each channel, and estimating chroma for each channel according to the weights; correcting a hue based on the white pixel value adjusted in the fine-adjusting step and the channel-specific chroma estimated in the channel-specific chroma estimating step; Including, The RGB channel average values ​​represent average values ​​of pixel values ​​corresponding to red, green, and blue color channels.

16. The step of estimating white pixel values ​​comprises: calculating a gradient value of the white pixels for each direction based on a difference value between pixels having different hues and a difference value between white pixels among source pixel data provided from the pixel array; calculating the RGB channel average values ​​according to the position of the center pixel among source pixel data provided from the pixel array; Including, 16. The method of claim 15, further comprising estimating a white pixel value based on the direction-specific white pixel value calculated in the direction-specific white pixel value calculation step and the RGB channel average value calculated in the RGB channel average value calculation step.

17. The step of estimating white pixel values ​​comprises:

17. The method of claim 16, further comprising estimating a white pixel value corresponding to the center pixel using a horizontal filter and a vertical filter when the center pixel is a green pixel, a red pixel, or a blue pixel.

18. The step of fine-tuning the white pixel value comprises:

20. The method of claim 17, wherein, when the center pixel is a green pixel, a red pixel, or a blue pixel, the white pixel value is fine-tuned using a gradient value of the white pixel value estimated in the step of estimating the white pixel value.

19. In the step of estimating the channel-specific chroma, 16. The method of claim 15, wherein the chroma array is calculated by subtracting the adjusted white pixel value from the source pixel data.

20. In the step of estimating the channel-specific chroma, assigning chroma weights according to chroma similarities for each channel centered on the chroma value of a center pixel in the chroma array, and estimating chromas for each channel according to the weights; The method of claim 19 , wherein the chroma similarity represents a degree of similarity between the chroma of a center pixel and the chroma of a surrounding pixel.

Images