Image signal processing method for detecting linear fixed mode noise and apparatus for performing the same

Abstract

A method for detecting linear fixed pattern noise (FPN) includes: extracting a plurality of pixel values from a first row of an ROI; calculating a luminance value of the first row based on the extracted pixel values; calculating a first score indicative of a variance of pixel values of the first row; calculating a second score indicating a difference between an average value of pixel values of the first row and an average value of pixel values of adjacent rows of the first row; and determining whether there is fixed pattern noise in the first row based on the first score and a first threshold and the second score and a second threshold.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10 - 2024 - 0004797, filed with the Korean Intellectual Property Office on January 11, 2024, the entire contents of which are incorporated herein by reference. Technical field

[0003] Various exemplary embodiments of the inventive concept relate to an electronic device including an image sensor, a method of operating the image sensor, and / or a system including the image sensor, etc., and more particularly, to an image signal processing method for detecting linear fixed - pattern noise, a device for performing the image signal processing method, and / or a system for performing the image signal processing method, etc. Background art

[0004] Types of image sensors include CCD (Charge - Coupled Device) image sensors, CMOS (Complementary Metal - Oxide - Semiconductor) image sensors (CIS), etc. A CMOS image sensor includes pixels composed of CMOS transistors and uses a photoelectric conversion element included in each pixel to convert light energy into at least one electrical signal. The CMOS image sensor uses the electrical signals generated in each pixel to obtain information related to the captured image.

[0005] Specifically, the main function of an automotive image sensor installed in a vehicle is video image capture rather than static image capture. Therefore, the automotive image sensor should ensure a frame rate above a specific level. When linear fixed - pattern noise (FPN) exists in the captured image in the form of vertical lines and / or horizontal lines, this may pose a threat to vehicle safety. Therefore, ISO26262 (International Standard for Functional Safety of Electrical / Electronic Systems in Vehicles) requires that such noise be detected. To this end, the noise detection function is usually implemented in hardware. However, a scheme for detecting noise through the implementation of a software algorithm of an SOC (System - on - Chip) is under research. Summary of the invention

[0006] Various exemplary embodiments of the inventive concept provide a method for real - time detecting linear fixed - pattern noise included in image data by an image signal processor existing outside an image sensor, a device including the image signal processor, and / or a system including the image signal processor, etc.

[0007] In addition, the technical advantages and / or objectives of one or more exemplary embodiments of the inventive concept are to provide a method for improving and / or optimizing power consumption by adjusting the calculation amount required for detecting linear fixed - pattern noise according to illuminance, a device for performing the method, and / or a system for performing the method, etc.

[0008] The technical advantages and / or purposes of the various exemplary embodiments of the inventive concept are not limited to the technical advantages and / or purposes described above, and other technical advantages and / or purposes not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

[0009] According to some exemplary embodiments of the inventive concept, a method for detecting linear fixed pattern noise is provided. The method includes: determining at least a part of image data received from an image sensor as a region of interest (ROI); extracting a plurality of pixel values from a first row of the ROI; calculating a luminance value of the first row based on the extracted pixel values; determining a first threshold related to a variance of the pixel values of the first row and a second threshold related to a difference between an average value of the pixel values of the first row and an average value of pixel values of an adjacent row of the first row based on the calculated luminance value; calculating a first score indicating the variance of the pixel values of the first row; calculating a second score indicating a difference between the average value of the pixel values of the first row and the average value of the pixel values of the adjacent row of the first row; and determining whether there is fixed pattern noise in the first row based on the first score and the first threshold and the second score and the second threshold.

[0010] According to at least one exemplary embodiment of the inventive concept, an image signal processor is provided. The image signal processor includes: a processing circuit configured to: process image data received from an image sensor; detect whether there is fixed pattern noise in each row of the processed image data; determine a part of the received processed image data as a region of interest (ROI); extract pixel values of each row of the ROI; calculate a luminance value of each row of the ROI based on the extracted pixel values; determine a first threshold related to a variance of the pixel values of each row of the ROI and a second threshold related to a difference between an average value of the pixel values of each row of the ROI and an average value of pixel values of an adjacent row of each row of the ROI based on the calculated luminance value; calculate a first score indicating the variance of the pixel values of each row of the ROI; calculate a second score indicating a difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of each row; and determine whether there is fixed pattern noise in each row of the ROI based on the first score and the first threshold and the second score and the second threshold.

[0011] According to at least one exemplary embodiment of the inventive concept, an electronic device is provided. The device includes: an image sensor including a pixel array including a plurality of pixels, each pixel of the plurality of pixels including a first photodiode and a second photodiode, the second photodiode having a larger light-receiving area than the first photodiode, wherein the pixel array is configured to: output a first pixel signal using the second photodiode based on a first conversion gain within a first illuminance range; output a second pixel signal using the second photodiode based on a second conversion gain within a second illuminance range; output a third pixel signal using the first photodiode based on the first conversion gain within a third illuminance range; and output a fourth pixel signal using the first photodiode based on the second conversion gain within a fourth illuminance range, wherein the first conversion gain is higher than the second conversion gain; the image sensor is configured to: perform sampling on each of the first pixel signal to the fourth pixel signal; and output image data based on the sampling result to a processing circuit; and the processing circuit is configured to: determine a part of the image data as a region of interest (ROI); extract pixel values of each row of the ROI; calculate luminance values of each row of the ROI based on the extracted pixel values; based on the calculated luminance values, determine a first threshold related to a variance of the pixel values of each row of the ROI, and determine a second threshold related to a difference between an average value of the pixel values of each row of the ROI and an average value of pixel values of an adjacent row of the ROI; calculate a first score indicating the variance of the pixel values of each row of the ROI; calculate a second score indicating a difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of an adjacent row of the ROI; and determine whether there is fixed pattern noise in each row of the ROI based on the first score and the first threshold and the second score and the second threshold.

[0012] According to at least one exemplary embodiment of the inventive concept, the method can detect linear fixed pattern noise and, thus, can reduce and / or stop false driving support based on a vehicle image sensor that malfunctions due to deterioration, thereby reducing and / or preventing vehicle passengers from being in a dangerous state.

[0013] In addition, according to at least one exemplary embodiment of the inventive concept, it is possible to reduce and / or eliminate the need to install a separate fault sensing circuit in the image sensor. Accordingly, the image sensor can be made smaller or further functions can be added.

[0014] In addition, according to at least one exemplary embodiment of the inventive concept, by adjusting the number of pixels for which calculation is expected and / or required based on illuminance, the amount of calculation performed can be reduced, thereby reducing power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other aspects and features of the inventive concept will become more apparent by describing in detail some example embodiments of the inventive concept with reference to the accompanying drawings, in which:

[0016] Figure 1 An example of the configuration of an image processing system according to at least one example embodiment of the inventive concept is shown;

[0017] Figure 2 An example of the configuration of an image sensor in Figure 1 according to some example embodiments is shown;

[0018] Figure 3 is a circuit diagram showing an example of a pixel in a pixel array in Figure 2 according to some example embodiments;

[0019] Figure 4 An example of the configuration of an image signal processor and a noise detection module in Figure 1 according to some example embodiments is shown;

[0020] Figure 5 An example of mapping a threshold of the variance of pixel values in corresponding rows in the horizontal direction to the luminance of image data, and an example of mapping a threshold of the difference between the average value of pixel values in a corresponding row and the average value of pixel values in an adjacent row to the luminance of image data are shown;

[0021] Figure 6 An example of code for fixed pattern noise detection according to at least one example embodiment of the inventive concept is shown;

[0022] Figure 7 An example of the detected fixed pattern noise in the horizontal direction according to at least one example embodiment of the inventive concept is shown;

[0023] Figure 8 Conceptually shows a linear fixed pattern noise detection operation according to at least one example embodiment of the inventive concept; and

[0024] Figure 9 is a flowchart showing an example of a linear fixed pattern noise detection method according to at least one example embodiment of the inventive concept. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the inventive concept will now be described with reference to the accompanying drawings. By referring to the following detailed descriptions of the respective exemplary embodiments and the drawings, the advantages and features of the exemplary embodiments of the inventive concept and methods of achieving them can be more easily understood. However, the exemplary embodiments of the inventive concept may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those of ordinary skill in the art, and the exemplary embodiments of the inventive concept are defined only by the appended claims.

[0026] When adding reference numerals to the components of the respective drawings, it should be noted that although the same components are shown in different drawings, the same reference numerals are assigned to the same components as much as possible. In addition, when determining that a detailed description of a related well-known configuration or function may obscure the gist of the exemplary embodiments of the inventive concept, its detailed description will be omitted.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in the sense commonly understood by those of ordinary skill in the art. In addition, terms defined in commonly used dictionaries should not be ideally or overly interpreted if they are not clearly specifically defined. The terms used herein are for the purpose of describing particular exemplary embodiments only and are not intended to limit the inventive concept. In this specification, the singular also includes the plural unless the statement clearly indicates otherwise.

[0028] In addition, when describing the components of the exemplary embodiments of the inventive concept, terms such as first, second, A, B, (a), (b) may be used. These terms are only used to distinguish a component from other components, and the nature or order of the component is not limited by the terms. If a component is described as "connected" to, "coupled" to, or "contacting" another component, the component may be directly connected to or in contact with the other component, but it should be understood that another component may also be "connected", "coupled", or "in contact" between each component.

[0029] The terms "comprising", "including", "having", etc. used in this specification specify the presence of the described features, integers, steps, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0030] Figure 1FIG. 0 shows an example of the configuration of an image processing system 10 according to at least one example embodiment of the inventive concept. For example, the image processing system 10 may be included as part of various electronic devices such as cameras, smartphones, wearable devices, Internet of Things (IoT) devices, home appliances, tablet devices, personal computers (PCs), personal digital assistants (PDAs), portable multimedia players (PMPs), personal navigation devices, drones, and / or advanced driver assistance systems (ADASs), etc. In addition, the image processing system 10 may be installed in an electronic device included as a component of a vehicle, furniture, manufacturing equipment, door, and / or various measurement devices, etc. For clarity and conciseness, it is assumed that the image processing system 10 according to at least one example embodiment of the inventive concept is installed in an electronic device provided as part of a vehicle, but the example embodiment is not limited thereto. Referring to Figure 1 , the image processing system 10 may include a lens 12, an image sensor 14, and / or an image signal processor 16, etc., but is not limited thereto.

[0031] Light (e.g., natural light, light from a flash, etc.) may be reflected from an object, a landscape, etc. that is an imaging target, and the reflected light may be received by the lens 12. The image sensor 14 may generate an electrical signal based on the light received through the lens 12. For example, the image sensor 14 may be implemented as a complementary metal oxide semiconductor (CMOS) image sensor, but is not limited thereto.

[0032] The image sensor 14 may include a pixel array. Pixels in the pixel array may convert the received light into an electrical signal to generate a pixel value. The ratio at which light is converted into an electrical signal (e.g., voltage) may be defined as a conversion gain. The pixel array may use dual conversion gain that changes the conversion gain to generate pixel signals in a low conversion gain mode and a high conversion gain mode. In addition, each pixel of the pixel array may have a split photodiode structure. Referring to Figure 2 The configuration of the image sensor 14 will be described in more detail.

[0033] The image signal processor 16 may be located outside the image sensor 14 and may perform preprocessing on the electrical signal output from the image sensor 14. The image signal processor 16 may appropriately process the preprocessed electrical signal to generate image data related to an imaging target (e.g., an object being imaged, a landscape, etc.). To this end, the image signal processor 16 may perform various processes such as color correction, automatic white balance, gamma correction, color saturation correction, bad pixel correction, and / or hue correction, etc. For example, the image signal processor 16 may perform processing on an image based on noise detected by a noise detection module (e.g., the noise detection module 18 discussed below) in the generated image data.

[0034] Specifically, an image signal processor 16 according to at least one example embodiment of the inventive concept may include, for example, a noise detection module 18 to detect linear fixed pattern noise (FPN) present on the generated image data, but is not limited thereto. Linear fixed pattern noise refers to pattern noise in the form of vertical lines and / or horizontal lines, and may occur due to abnormalities, noise, etc. in the horizontal direction signal lines and / or vertical direction signal lines inside the image sensor 14. According to at least one example embodiment of the inventive concept, such linear fixed pattern noise may be detected in real time by the noise detection module 18 of the image signal processor 16, but is not limited thereto. Hereinafter, for convenience of description, it is assumed that the noise represents linear fixed pattern noise. According to some example embodiments, one or more of the image signal processor 16 and / or the noise detection module 18, etc. may be implemented as a processing circuit. The processing circuit may include hardware or a hardware circuit including a logic circuit; a hardware / software combination, such as a processor executing software and / or firmware; or a combination thereof. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc., but is not limited thereto.

[0035] Figure 1 One lens 12 and one image sensor 14 are shown. However, in at least one other example embodiment, the image processing system 10 may include multiple lenses and / or multiple image sensors. In this case, the multiple lenses may each have a different viewing angle. In addition, the multiple image sensors may each have different functions, different performances, and / or different characteristics, and may each include pixel arrays with different configurations.

[0036] Figure 2 shows Figure 1 an example of the configuration of the image sensor 14 in. Refer to Figure 2, the image sensor 100 may include a pixel array 110, a row driver 120, a ramp signal generator 130, an analog-to-digital converter (ADC) circuit 140, a data bus 150, an enable signal generator 160, and / or a timing controller 170, etc., but not limited thereto. According to some example embodiments, one or more of the image sensor 14, the row driver 120, the ramp signal generator 130, the analog-to-digital converter (ADC) circuit 140, the data bus 150, the enable signal generator 160, and / or the timing controller 170, etc. may be implemented as a processing circuit. The processing circuit may include hardware or a hardware circuit including logic circuits; a hardware / software combination, such as a processor executing software and / or firmware; or a combination thereof. For example, the processing circuit may more specifically include, but not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc., but not limited thereto.

[0037] The pixel array 110 may include a plurality of pixels PX arranged in a matrix form (i.e., an N×N configuration) along N rows and N columns, where N is an integer, but the example embodiments are not limited thereto. Each of the plurality of pixels PX may include a photoelectric conversion element. For example, the photoelectric conversion element may include a photodiode, a phototransistor, a photogate, and / or a pinned photodiode, etc. In addition, each of the plurality of pixels PX may include a plurality of photoelectric conversion elements.

[0038] Each of the plurality of pixels PX according to at least one example embodiment of the inventive concept may be a pixel PX having a split photodiode structure including at least two or more photodiodes, but not limited thereto. In this regard, two or more photodiodes may operate independently of each other. For example, the pixel PX may include a small photodiode (SPD) having a smaller light-receiving area and a large photodiode (LPD) having a light-receiving area larger than that of the SPD, but not limited thereto.

[0039] The large photodiode and the small photodiode may be selectively operated according to and / or based on the illuminance, etc. around an object (e.g., a target object). For example, the large photodiode may operate in a low illuminance environment to generate a pixel signal, while the small photodiode may operate in a high illuminance environment to extend the exposure time to generate a pixel signal, but the example embodiments are not limited thereto. In addition, each of the large photodiode and the small photodiode may operate in a high conversion gain mode and / or a low conversion gain mode. Refer to Figure 3 The configuration and operation of the pixel PX having a split photodiode structure are described in more detail.

[0040] In at least one exemplary embodiment, the microlenses for light concentration may be located on top of each of the plurality of pixels PX, and / or on top of each pixel group composed of adjacent pixels PX. Each of the plurality of pixels PX may detect light in a specific spectral region in the light received through the microlenses. For example, the pixel array 110 may include red pixels that convert light in the red spectral region into an electrical signal, green pixels that convert light in the green spectral region into an electrical signal, and blue pixels that convert light in the blue spectral region into an electrical signal, etc., but the exemplary embodiments are not limited thereto. The color filter may be located on top of each of the plurality of pixels PX and may transmit light in a specific spectral region therethrough. However, the exemplary embodiments of the inventive concept are not limited thereto, and the pixel array 110 may include pixels that convert light in other spectral regions than the red, green, and blue (RGB) spectral regions into an electrical signal (e.g., cyan, magenta, yellow, and black (CMYK) pixels, etc.).

[0041] Each of the plurality of pixels PX of the pixel array 110 may output a pixel signal based on the intensity and / or amount of light received from an external source via a corresponding one of the column lines CL1 to CLN. Each of the plurality of column lines CL1 to CLN may extend in the column direction and may be connected to the pixels PX arranged in the same column. For example, the pixel signal may be an analog signal corresponding to the intensity and / or amount of light received from the external source. The pixel signal may pass through a voltage buffer (e.g., a source follower, etc.) and then be provided to the ADC circuit 140 via the column lines CL1 to CLN.

[0042] The row driver 120 may select and drive the rows of the pixel array 110. The row driver 120 may decode the address and / or control signals generated by the timing controller 170 and generate control signals for selecting and driving the rows of the pixel array 110 based on the decoding result. For example, the control signals may include a signal for selecting a pixel, a signal for resetting the floating diffusion region, and / or a signal for transferring charge to the floating diffusion region, etc.

[0043] The ramp signal generator 130 may generate a ramp signal under the control of the timing controller 170. For example, the ramp signal generator 130 may operate under a control signal (e.g., a ramp enable signal). When the ramp enable signal is activated, the ramp signal generator 130 may generate a ramp signal based on an expected value and / or a predetermined value (e.g., a start level, an end level, a slope, etc.). In other words, the ramp signal may be a signal that increases and / or decreases according to an expected slope and / or a predetermined slope at a specific time. The ramp signal may be provided to the ADC circuit 140.

[0044] The ADC circuit 140 can receive pixel signals of a plurality of pixels PX from the pixel array 110 via the column lines CL1 to CLN, and the ADC circuit 140 can receive a ramp signal from the ramp signal generator 130. The ADC circuit 140 can obtain a reset signal and / or an image signal from the received pixel signals, and can extract the difference therebetween as a valid signal component. The ADC circuit 140 can include a plurality of comparators COMP and a counter CNT, but is not limited thereto.

[0045] Specifically, the comparator COMP can perform sampling by comparing the reset signal of the pixel signal with the ramp signal RAMP with each other and comparing the image signal of the pixel signal with the ramp signal RAMP with each other. For example, each comparator COMP can be implemented as an OTA (operational transconductance amplifier), but the exemplary embodiments are not limited thereto. The counter CNT can count the pulses of the signal that has undergone correlated double sampling, and output the counting result as a digital signal, and then can provide the digital signal to the data bus 150, but is not limited thereto.

[0046] The data bus 150 can output image data IDAT based on the digital signal received from the ADC circuit 140. For example, the data bus 150 can include a plurality of memories (e.g., storage devices, etc.), sense amplifiers, and / or column decoders, etc. The plurality of memories can temporarily store the digital signal output from the counter CNT. The sense amplifier can sense and amplify the digital signal stored in the plurality of memories. The operation of storing the digital signal in the plurality of memories and the operation of obtaining the stored digital signal therefrom can be performed under the control of the enable signal generator 160. The amplified digital signal can be sent to the Figure 1 image signal processor 16 in, but is not limited thereto.

[0047] The enable signal generator 160 can generate read / write enable signals and read / write select signals for controlling the operation of temporarily storing the digital signal in the plurality of memories included in the data bus 150 (i.e., the write operation) and the operation of obtaining the digital signal from the plurality of memories to output the image data IDAT (i.e., the read operation), but the exemplary embodiments are not limited thereto.

[0048] The timing controller 170 can generate control signals and / or clocks for controlling the operation and / or timing of each of the row driver 120, the ramp signal generator 130, the ADC circuit 140, and / or the enable signal generator 160, etc.

[0049] In at least one exemplary embodiment, when a defect occurs in column lines CL1 to CLN that are signal lines in the horizontal direction and / or signal lines in the vertical direction and are connected to each row of the pixel array 110, fixed pattern noise (FPN) in the horizontal direction and / or vertical direction may appear on the image data IDAT output from the data bus 150. The noise thus generated can be sent to Figure 1 the image signal processor 16 therein and be detected.

[0050] Figure 3 is a circuit diagram showing an example of one pixel PX of the pixel array 110 in Figure 2 . Referring to Figure 3 , the pixel PX may include a large photodiode LPD, a small photodiode SPD, a large transfer transistor LTG, a small transfer transistor STG, a reset transistor RG, a driving transistor DX, a selection transistor SX, a conversion gain control transistor DRG, a switching transistor SW, a capacitor control transistor CCTR, and / or a capacitor C1, etc., but is not limited thereto.

[0051] In addition, referring to Figure 3 , the voltages applied to the pixel PX may include a pixel voltage VPIX, a capacitor power supply voltage VMIM, and / or a reset power supply voltage VRD, etc. Each of the capacitor power supply voltage VMIM and the reset power supply voltage VRD may be supplied together with the pixel voltage VPIX. Additionally and / or alternatively, each of the capacitor power supply voltage VMIM, the reset power supply voltage VRD, and the pixel voltage VPIX may be supplied through separate circuits. In addition, parasitic capacitors may be generated by a plurality of floating diffusion regions (e.g., floating diffusion regions FD1, FD2, and FD3, etc.).

[0052] The photodiode can convert light incident from an external source into an electrical signal. The photodiode can generate charge according to and / or based on the intensity of the received light. According to the illuminance around the target object, the amount of charge generated in the photodiode can change. As described above, according to the size of the light-receiving area, the photodiode can be classified into a large photodiode LPD having a large light-receiving area and a small photodiode SPD having a small light-receiving area, but is not limited thereto. That is, the pixel PX may have a split photodiode structure including a large photodiode LPD and a small photodiode SPD, but is not limited thereto.

[0053] The large transfer transistor LTG can operate based on a large transfer control signal LTS. For example, the large transfer transistor LTG can transfer the charge generated by the large photodiode LPD to the third floating diffusion region FD3 or the like. In addition, according to some exemplary embodiments, when the conversion gain control transistor DRG is turned on, the large transfer transistor LTG can transfer not only the charge generated by the large photodiode LPD to the third floating diffusion region FD3, but also to another floating diffusion region (e.g., the second floating diffusion region FD2 or the like). One end of the large transfer transistor LTG can be connected to the large photodiode LPD, and the other end can be connected to the third floating diffusion region FD3 or the like.

[0054] The small transfer transistor STG can operate based on a small transfer control signal STS. The small transfer transistor STG can transfer the charge generated by the small photodiode SPD to, for example, the first floating diffusion region FD1 or the like. One end (e.g., the first end) of the small transfer transistor STG can be connected to the small photodiode SPD, and the other end (e.g., the second end) can be connected to the first floating diffusion region FD1, but is not limited thereto.

[0055] The switch transistor SW can operate based on a switch control signal SWS. The switch transistor SW can be turned on to generate a pixel signal PIX using the small photodiode SPD. The switch transistor SW can be turned off to generate a pixel signal PIX using the large photodiode LPD. One end (e.g., the first end) of the switch transistor SW can be connected to, for example, the first floating diffusion region FD1, and the other end (e.g., the second end) can be connected to, for example, the second floating diffusion region FD2, but is not limited thereto.

[0056] When using the large photodiode LPD, the conversion gain control transistor DRG can operate based on a conversion gain control signal CGS. When the conversion gain control transistor DRG is turned on, the parasitic capacitance generated in the third floating diffusion region FD3 can be in parallel with the parasitic capacitance generated in the second floating diffusion region FD2, and thus, the capacitance of the floating diffusion region can be increased. When the capacitance of the floating diffusion region increases, the conversion gain decreases, and when the capacitance of the floating diffusion region decreases, the conversion gain increases. Therefore, the conversion gain when the conversion gain control transistor DRG is turned off can be higher than the conversion gain when the conversion gain control transistor DRG is turned on. One end (e.g., the first end) of the conversion gain control transistor DRG can be connected to the second floating diffusion region FD2, and the other end (e.g., the second end) can be connected to the third floating diffusion region FD3 or the like.

[0057] When using a small photodiode SPD, the capacitor control transistor CCTR can operate based on a capacitor control signal CCS. When the capacitor control transistor CCTR is turned on, the capacitor C1 can be connected in parallel with the parasitic capacitance generated in the first floating diffusion region FD1 to increase the capacitance of the first floating diffusion region FD1. Therefore, the conversion gain when the capacitor control transistor CCTR is turned off can be higher than the conversion gain when the capacitor control transistor CCTR is turned on. One end (e.g., the first end) of the capacitor control transistor CCTR can be connected to the capacitor C1, and the other end (e.g., the second end) can be connected to the capacitor voltage VMIM. For example, the capacitor C1 can be a passive element with a fixed and / or variable capacitance and can store the lateral overflow charge from the small photodiode SPD.

[0058] The reset transistor RG can operate based on a reset control signal RS and can supply a reset power supply voltage VRD to, for example, the floating diffusion regions FD2 and FD3. In addition, when the switch transistor SW is turned on, the reset transistor RG can also supply the reset power supply voltage VRD to, for example, the first floating diffusion region FD1. As a result, the charges accumulated in the floating diffusion regions FD1, FD2, and FD3 can migrate to the reset power supply voltage VRD terminal. The voltages of the floating diffusion regions FD1, FD2, and FD3 can be reset.

[0059] The driving transistor DX can operate as a source follower based on a bias current generated by a current source (not shown) connected to the column line CL and can amplify the voltages of the floating diffusion regions FD1, FD2, and FD3 to generate a pixel signal PIX. The selection transistor SX can operate based on a selection signal SEL and can select the pixels to be read row by row. When the selection transistor SEL is turned on, the pixel signal PIX can be output to the Figure 2 ADC circuit 140 in, but the exemplary embodiments are not limited thereto.

[0060] Therefore, Figure 3 the pixel PX in can generate the pixel signal PIX using a large photodiode LPD or a small photodiode SPD. In addition, the large photodiode LPD can operate in a high conversion gain mode or a low conversion gain mode depending on whether the conversion gain control transistor DRG is turned on or off. The small photodiode SPD can operate in a high conversion gain mode or a low conversion gain mode depending on whether the capacitor control transistor CCTR is turned on or off.

[0061] In other words, Figure 3The pixels PX therein can generate pixel signals PIX in a total of four readout modes according to the illuminance. Specifically, in the first range with the lowest illuminance, the large photodiode LPD can operate in a high conversion gain mode (hereinafter referred to as the LPD_HCG mode). In the second range with an illuminance higher than that of the first range, the large photodiode LPD can operate in a low conversion gain mode (hereinafter referred to as the LPD_LCG mode). In the third range with an illuminance higher than that of the second range, the small photodiode SPD can operate in a high conversion gain mode (hereinafter referred to as the SPD_HCG mode). In the fourth range with the highest illuminance, the small photodiode SPD can operate in a low conversion gain mode (hereinafter referred to as the SPD_LCG mode). However, the exemplary embodiments are not limited thereto. For example, the image sensor can have a greater or fewer number of illuminance ranges and / or gain modes.

[0062] In this way, the pixels PX can use the large photodiode LPD and the small photodiode SPD that can operate in a dual conversion gain mode to detect low and high amounts of light according to the illuminance. Therefore, Figure 2 the dynamic range of the image sensor 100 therein can be increased. In addition, the pixels PX can operate in the LPD_HCG mode, the LPD_LCG mode, the SPD_HCG mode, and the SPD_LCG mode in sequence. Figure 2 The image sensor 100 therein can combine all the image data IDAT corresponding to the four modes with each other to generate an HDR (high dynamic range) image, but is not limited thereto.

[0063] Figure 4 shows an example of the configuration of the Figure 1 image signal processor 16 and the noise detection module 18 according to some exemplary embodiments. Referring to Figure 4 , the image signal processor 200 can include an image processing module 210, a noise detection module 220, and / or an output module 230, etc., but is not limited thereto. The noise detection module 220 can include an image input module 221, an operation module 222, a statistical module 223, and / or a result storage module 224, etc., but is not limited thereto. In at least one exemplary embodiment, the components (e.g., modules) shown in Figure 4 represent functionally different functional elements, but are not limited thereto. Therefore, it should be understood that at least two components (e.g., modules) can be implemented in an integrated form in an actual physical environment. The following will refer to Figure 2 as well as Figure 4A description will be given. According to some example embodiments, one or more of the image signal processor 200, the image processing module 210, the noise detection module 220, the output module 230, the image input module 221, the operation module 222, the statistical module 223, and / or the result storage module 224, etc. can be implemented as a processing circuit. The processing circuit can include hardware or a hardware circuit including logic circuits; a hardware / software combination, such as a processor executing software and / or firmware; or a combination thereof. For example, the processing circuit can more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc., but is not limited thereto.

[0064] The image processing module 210 can receive the image data IDAT input from the image sensor 100 into the image signal processor 200. The image processing module 210 can perform various processes on the image data IDAT, where the various processes can include color correction, automatic white balance, gamma correction, color saturation correction, bad pixel correction, and / or hue correction, etc. In addition, the image processing module 210 can extract information required to support the vehicle driver and / or necessary information from the image data IDAT, but the example embodiments are not limited thereto. The image processing module 210 can send the processed image data IDATˋ to the noise detection module 220. In the following description, in order to distinguish the processed image data IDATˋ from the image data IDAT output from the image sensor 100, the image data processed in the image processing module 210 will be represented as IDATˋ.

[0065] The noise detection module 220 may detect fixed pattern noise (FPN) in the horizontal direction and / or the vertical direction from the image data IDAT' provided by the image processing module 210. For clarity and conciseness, the following description will focus on at least one exemplary embodiment, in which the noise detection module 220 detects fixed pattern noise in the horizontal direction (e.g., row direction), and it is assumed that the linear fixed pattern noise is the fixed pattern noise in the horizontal direction, but the exemplary embodiment is not limited thereto, and for example, the fixed pattern noise may be the fixed pattern noise in the vertical direction (e.g., column direction), or the fixed pattern noise in both the horizontal and vertical directions. First, the image data IDAT' processed in the image processing module 210 may be input through the image input module 221. Subsequently, the operation module 222 sets a partial area of the image data IDAT' as the ROI (region of interest), and then the variance of the pixel values of each row of the ROI may be compared with an expected threshold and / or a preset threshold, and the difference between the average pixel value of each corresponding row and the average pixel value of the adjacent row may be compared with the expected threshold and / or the preset threshold, and based on the comparison result, it is determined whether there is fixed pattern noise. In this regard, the average value of the pixel values and the variance of the pixel values may be calculated by the statistical module 223. The detection result of the linear fixed pattern noise may be stored in the result storage module 224. The noise detection module may notify the output module 230 whether the corresponding image data IDAT' includes fixed pattern noise. The operation of the operation module 222 will be described in more detail below.

[0066] All pixels in a row in which linear fixed pattern noise is detected have the lowest pixel value or the highest pixel value among the pixel values of the image data IDATˋ. In other words, the variance of the pixel values in the row in which linear fixed pattern noise is detected is less than and / or much less than the variance of the pixel values in each other row. Moreover, since the pixels in the row in which linear fixed pattern noise is detected have the lowest pixel value or the highest pixel value, the difference between the average value of the pixel values in the row in which linear fixed pattern noise is detected and the average value of the pixel values in the adjacent row thereto is relatively large compared to a target row (e.g., the row being analyzed), or in other words, the difference between the average value of the pixel values in the target row and the average value of the pixel values in the adjacent row of the target row exceeds an expected threshold. In addition, since such a trend may also appear in a part (e.g., a subset) of the row in which linear fixed pattern noise is detected, the above-described detection operation may be performed on a ROI on a part (e.g., a subset) of the image data IDATˋ instead of the entire image data IDATˋ. Moreover, when the detection operation is performed on a ROI on a part of the image data IDATˋ, the amount of computation for performing image analysis can be reduced, and thus the power consumed by the image analysis can be reduced. Therefore, if the variance of the pixel values in a row in the ROI of the image data IDATˋ is less than or equal to an expected threshold and / or a predetermined threshold, and the difference between the average value of the pixel values in the row and the average value of the pixel values in the adjacent row of the row is greater than or equal to an expected threshold and / or a predetermined threshold, the noise detection module 220 may determine that there is fixed pattern noise in the horizontal direction in the row.

[0067] Specifically, the operation module 222 may set a partial area of the input image data IDATˋ as a ROI. The ROI may vary according to the circuit configuration of the image sensor 100, but is not limited thereto. When the row driver 120 for controlling the horizontal direction is located, for example, on the left side of the pixel array 110, noise may appear from the left end of the pixel array 110 and / or appear at the left end of the pixel array 110. In this case, the ROI may be set as the left partial area of the image data IDATˋ. On the other hand, when the row driver 120 is located on the right side of the pixel array 110, noise appears from the right end of the pixel array 110. In this case, the ROI may be set as the right partial area of the image data IDATˋ. In this regard, since the noise detection operation should be performed on all rows, the horizontal length of the ROI is equal to the horizontal length of the image data IDAT`, and the vertical length of the ROI may be set as a part (e.g., 1 / 6) of the vertical length of the image data IDAT`, but the exemplary embodiment is not limited thereto. That is, generally, the length of the ROI in the first direction may be set to be less than the length of the image data in the first direction, and the length of the ROI in the second direction perpendicular to the first direction may be set to be equal to the length of the image data in the second direction, etc.

[0068] First, the operation module 222 can extract the pixel values of each row of the pixel array 110 starting from the top row of the ROI, and calculate its luminance value based on the extraction result. When the number of pixels in the Nth row in the ROI is M N , the sum of the pixel values of the pixels in the Nth row is S N , and the total square of the pixel values in the Nth row is SS N , the luminance value Luma of the Nth row can be calculated based on the following mathematical formula 1 N :

[0069]

Mathematical formula 1

[0070] Luma N = Max((S N-1 + S N + S N+1 ) / (3 × M N ) - pedestal, O)

[0071] where pedestal can represent the pedestal value added to the image data IDAT`. Based on the luminance value calculated in this way, the variance threshold hor_th related to the variance of the pixel values of the Nth row in the horizontal direction and the difference threshold diff_th related to the difference between the average value of the pixel values of the Nth row and the average value of the pixel values of the adjacent row of the Nth row can be calculated. The threshold can vary flexibly according to and / or based on the luminance value of the image data IDAT`. The threshold can be mapped to the luminance value according to and / or based on the characteristics of the image sensor 100. This will be discussed in more detail in conjunction with Figure 5 .

[0072] Figure 5 shows examples of mapping the threshold of the variance of the pixel values of the corresponding row in the horizontal direction to the luminance value of the image data IDAT`, and examples of mapping the threshold of the difference between the average value of the pixel values of the corresponding row and the average value of the pixel values of the adjacent row of the corresponding row to the luminance value of the image data IDAT`. Referring to Figure 5 , when the luminance value changes to h1, h2, h3, and h4, the variance threshold hor_th of the pixel values of the corresponding row in the horizontal direction changes to h_th1, h_th2, and h_th3, but the exemplary embodiments are not limited thereto. When the luminance value changes to d1, d2, d3, and d4, the difference threshold diff_th between the average value of the pixel values of the corresponding row and the average value of the pixel values of the adjacent row of the corresponding row changes to d_th1, d_th2, and d_th3, but the exemplary embodiments are not limited thereto. This mapping relationship between the luminance value and each threshold can be required and / or predetermined according to and / or based on the characteristics of the image sensor 100, as described above, but is not limited thereto.

[0073] Return reference Figure 4 , the horizontal score hor_score of the Nth row to be compared with the threshold hor_th and the difference diff between the average value of the pixel values of the Nth row to be compared with the threshold diff_th and the average value of the pixel values of the adjacent rows (the (N - 1)th row and the (N + 1)th row) of the Nth row can be calculated based on the following mathematical formula 2 upper and diff lower :

[0074]

Mathematical formula 2

[0075]

[0076] The horizontal score hor_score can be obtained by dividing the variance of the pixel values of the Nth row by the average value of the pixel values of the Nth row, and can correspond to a coefficient of variation. In addition, diff upper can represent the difference between the average value of the pixel values of the Nth row and the average value of the pixel values of the (N - 1)th row, and diff lower can represent the difference between the average value of the pixel values of the Nth row and the average value of the pixel values of the (N + 1)th row. Additionally, the horizontal score, diff upper and diff lower values can be calculated based on a base value and / or a value based on image compensation (e.g., a compensator).

[0077] In at least one example embodiment, when performing a noise detection operation on the first row, the value diff upper will not be calculated. When performing a noise detection operation on the last row, the value diff lower will not be calculated. As described above, the variance and average value of the pixel values can be calculated in the statistical module 223. However, according to specific embodiments, the variance and average value of the pixel values can be calculated in, for example, the image processing module 210 and / or can be calculated in the operation module 222, etc. The thresholds, horizontal scores, and / or differences diff upper and diff lower calculated in this way can be compared with each other, as Figure 6 shown, but the example embodiments are not limited thereto.

[0078] Figure 6 shows an example of code (e.g., software, computer-readable instructions, etc.) for fixed pattern noise detection according to at least one example embodiment of the inventive concept. Refer to Figure 6, the image processing module 210 may first determine whether the Nth row for which the noise detection operation is currently performed is the first row or the last row, but is not limited thereto. If the Nth row is the first row, the value diff is not calculated upper , so only the value diff lower and hor_score are compared with their respective thresholds. If the Nth row is the last row, the value diff is not calculated lower , so only the value diff upper and hor_score are compared with their respective thresholds. If the Nth row is neither the first row nor the last row, the values diff upper , diff lower and hor_score are compared with their respective thresholds. Specifically, if each of the values diff upper and the value diff lower is greater than or equal to the value diff_th(luma) as the threshold, and the value hor_score is less than or equal to the hor_th(luma) as the threshold, the image processing module 210 may determine that there is fixed pattern noise in the horizontal direction in the Nth row. In this regard, each of diff_th(luma) and hor_th(luma) indicates that the threshold is a function of the luminance value. If fixed pattern noise is detected in the Nth row, 1 is recorded in the Nth row of the result matrix diagram to store the noise detection result, and if no fixed pattern noise is detected in the Nth row, 0 is recorded, but is not limited thereto. This result matrix diagram may be stored in the result storage module 224. Subsequently, the above operations will be performed on all rows of the ROI.

[0079] Return reference Figure 4 , the output module 230 may output the image data IDAT` processed in the image processing module 210, etc. Then, the output module 230 may output relevant information based on the detection result of the noise detection module 220. Specifically, the result storage module 224 may provide the output module 230 with the noise detection result including the result matrix. The output module 230 may accordingly output information indicating that there is noise in the image data IDAT` as the output, etc. For example, if noise is detected in at least one exemplary embodiment where the image sensor 100 and the image signal processor 200 are installed in a vehicle, a message indicating that noise is detected in the image data IDAT` may be output to the vehicle. In addition, the image signal processor 200 may correct and / or compensate for the noise detected and / or identified in the image data IDAT` using noise correction techniques based on the noise detection result.

[0080] At least one example embodiment of detecting noise using all pixel values of corresponding rows has been described above. However, in some cases, pixel values of at least one specific color may be used instead of all pixel values. For example, when the pixel array 110 uses an RGB color filter, green pixel values may be used as representative pixel values. When the pixel array 110 uses a CMYK color filter, yellow pixel values may be used as representative pixel values.

[0081] In this way, the number of pixels used in the noise detection operation may vary according to and / or based on the illuminance around the object. For example, within the first range to the second range having low illuminance (i.e., the low illuminance range using the large photodiode LPD) described above, all pixel values may be used to detect noise, but the example embodiment is not limited thereto. Within the third range to the fourth range having high illuminance (i.e., the high illuminance range using the small photodiode SPD), noise can be sufficiently detected even if only some pixel values are used. That is, according to at least one example embodiment of the inventive concept, when performing the noise detection operation, the image signal processor 200 may determine whether to use all pixels of each row or only some of them based on the illuminance value. Therefore, the amount of computation required and / or desired for the noise detection operation can be adjusted according to the illuminance value. Accordingly, power consumption can be reduced and / or optimized. Figure 3 At least one example embodiment of detecting fixed pattern noise in the horizontal direction using the above operations has been described above. In at least one example embodiment, fixed pattern noise in the vertical direction (e.g., column direction) can be detected by rotating the image data IDATˋ by 90 degrees before setting the ROI, and then performing the same operations as described above (e.g., using the image processing module 210, etc.) on the rotated image data IDATˋ. In other words, since the image data IDATˋ is rotated by 90 degrees, the columns of the original image data IDATˋ become the rows of the rotated image data IDATˋ, so this method operation can be used to detect fixed pattern noise in the columns of the original image data IDATˋ. Through this noise detection operation, circuit anomalies, errors, and / or noises inside the image sensor 100 can be detected, and it is not necessary and / or not required to install a separate fault detection circuit, etc. inside the image sensor 100. Therefore, the image sensor can be designed to have a smaller physical area, and / or other functions other than fault detection can be added thereto. Thereby, malfunctions of the driving assistance function and / or the autonomous driving function of the vehicle can be reduced and / or prevented, and thus the safety of vehicle passengers can be improved and / or enhanced.

[0082]

[0083] Figure 7 Figure 7An example of the detected fixed - pattern noise in the horizontal direction according to at least one example embodiment of the inventive concept is shown. This linear fixed - pattern noise is output as a HI digital signal or a LO digital signal. Thus, a row will be filled with the highest pixel value or the lowest pixel value in the image data. Even if a complete HI signal or LO signal cannot be output due to a calculation error inside the image sensor, a row is still filled with the highest pixel value or the lowest pixel value. In other words, a row with linear fixed - pattern noise has a small variance and / or a very small variance compared to the variance of the pixel values of each other row where no noise is detected. When adjacent rows of a row with linear fixed - pattern noise consist of normal pixels and / or include normal pixels, the difference between the average pixel value of the row with linear fixed - pattern noise and the average pixel value of its adjacent rows is relatively large compared to the target row (e.g., the row being analyzed), or in other words, the difference between the average pixel value of the target row and the average pixel value of its adjacent rows exceeds a desired threshold. In this way, fixed - pattern noise in the horizontal direction can be detected. In some example embodiments, although Figure 7 not shown, the same principle can be used to detect fixed - pattern noise in the vertical direction.

[0084] Figure 8 Conceptually shows the linear fixed - pattern noise detection operation according to at least one example embodiment of the inventive concept. Assume Figure 8 the image data IDATˋ shown is output from an image sensor using a CMYK color filter, and its horizontal length is W and vertical length is H, but the example embodiment is not limited thereto. Assume that an ROI (region of interest) is set to have a horizontal length of W / 6 starting from the left end of the image data IDATˋ, but it is not limited thereto. The noise detection operation according to at least one example embodiment of the inventive concept will be performed on all rows of the ROI, where the (N - 1)th row and the (N + 1)th row adjacent to the Nth row are as Figure 8 shown etc.

[0085] Figure 8 shows that the noise detection operation is performed only on representative pixels instead of all pixels of each row. However, the example embodiment is not limited thereto. In this case, since it is assumed that a CMYK color filter is used, the representative pixel is a yellow pixel, but it is not limited thereto. As Figure 8 shown, when the noise detection operation is performed only using representative pixels (i.e., yellow pixels), the calculation will be performed only on L / 2 which is half of the horizontal length L = W / 6 of the ROI, but the example embodiment is not limited thereto. As referred to above Figures 4 to 6As described above, noise detection operations can be performed by comparing the horizontal score of the Nth row with a relevant threshold, and by comparing the difference between the average value of the pixel values of the Nth row and the average values of the pixel values of the adjacent (N-1)th and (N+1)th rows with a relevant threshold, etc. Figure 8 Illustrates a case where fixed pattern noise in the horizontal direction is detected in the Nth row. In this case, a value such as 1 can be recorded in map[N] of the Nth row of the matrix diagram in which the detection result can be stored.

[0086] Figure 9 Is a flowchart showing an example of a linear fixed pattern noise detection method according to at least one example embodiment of the inventive concept. For reference, Figure 9 Illustrates the operations performed in Figure 4 The image signal processor 200 in, but the example embodiment is not limited thereto. Therefore, it can be understood that in the following description, when the subject of a specific operation is omitted, the relevant operation can be performed in Figure 4 The image signal processor 200 in, but not limited thereto. The method will be described below with reference to Figure 9 And Figures 4 to 6 But the example embodiment is not limited thereto.

[0087] In operation S100, the image signal processor 200 may determine at least a part (e.g., a subset, etc.) of the image data received from the image sensor 100 as a region of interest (ROI). The length of the ROI in the first direction may be set to be less than the length of the image data in the first direction, and the length of the ROI in the second direction perpendicular to the first direction may be set to be equal to the length of the image data in the second direction, but the example embodiment is not limited thereto. In operation S200, the image signal processor 200 may calculate the luminance value of a row (e.g., the first row) of the image data by extracting the pixel values of the first row of the ROI. In operation S300, based on the calculated luminance value, the image signal processor 200 may determine a first threshold hor_th related to the variance of the pixel values of the first row, and a second threshold diff_th related to the difference between the average value of the pixel values of the first row and the average values of the pixel values of the adjacent rows of the first row. These thresholds may be mapped by the image signal processor 200 to specific luminance values according to and / or based on the characteristics of the image sensor 100, as described with reference to Figure 5 But not limited thereto.

[0088] In operation S400, the image signal processor 200 may calculate a first score hor_score and a second score diff upper And diff lower , where the first score hor_score indicates the variance of the pixel values of the first row, and the second score diffupper and diff lower respectively indicate the difference between the average value of the pixel values of the first row and the average value of the pixel values of each of the adjacent rows of the first row. The first score (i.e., the horizontal score hor_score) can correspond to the variance of the pixel values of the first row divided by the average value of the pixel values of the first row, etc. The second score can correspond to the difference between the average value of the pixel values of the first row and the average value of the pixel values of the adjacent row of the first row. Specifically, if the first row is the first row of the image data, only the value diff lower etc. can be calculated. If the first row is the last row of the image data, only the value diff upper etc. can be calculated.

[0089] In operation S500, if the first score is less than or equal to the first threshold (hor_score <= hor_th), and the second score is greater than or equal to the second threshold (diff upper >= diff_th, diff lower >= diff_th), then the image signal processor 200 can determine that there is fixed pattern noise in the horizontal direction in the first row. In operation S600, the image signal processor 200 can display information indicating the presence of fixed pattern noise in the first row along with the output of the image data, etc. In some example embodiments, when detecting fixed pattern noise in the vertical direction, the image signal processor 200 can rotate the received image data by 90 degrees before the operation S100 of setting the ROI, and then, the same operations as described above can be performed on the rotated data, but the example embodiments are not limited thereto.

[0090] According to at least one example embodiment of the inventive concept, the method can detect linear fixed pattern noise and correct and / or compensate for the linear fixed pattern noise, thereby reducing and / or stopping incorrect driving support based on images generated using a faulty and / or noisy image sensor of a vehicle that has malfunctioned due to deterioration or the like (e.g., detection of pedestrians, vehicles, and / or obstacles in an image, detection of traffic signals in an image, generation of an autonomous driving instruction based on the analyzed image, etc.), thereby reducing and / or preventing vehicle passengers from being in a dangerous state. In other words, the corrected image can be beneficial to an autonomous driving vehicle because it reduces variations in which noise generated by a faulty image sensor negatively affects image analysis (e.g., pedestrian detection, other vehicle detection, obstacle detection, traffic signal detection, etc.) performed by the autonomous driving vehicle, thereby improving the safety and reliability of the autonomous driving vehicle. In addition, according to at least one example embodiment of the inventive concept, since a fault (e.g., image noise, etc.) can be detected and / or corrected in real time while the vehicle is in motion, incorrect autonomous driving instructions based on noisy and / or faulty images can be reduced and / or eliminated because noise in the image can be corrected and / or compensated more quickly by the vehicle's image signal processor. In addition, there is no need to install a separate fault detection circuit in the image sensor. Therefore, the physical size of the image sensor can be smaller, and / or more functions can be added. In addition, according to at least one example embodiment of the inventive concept, the amount of computation used by the image sensor can be reduced by calculating the required and / or desired number of pixels based on the illuminance adjustment, thereby reducing the power consumption of the image sensor. In addition, the method for detecting linear fixed pattern noise according to at least one example embodiment of the inventive concept can be implemented using a software-based algorithm without adding a separate hardware device. Therefore, the cost of the image sensor can also be reduced.

[0091] Although various example embodiments of the inventive concept have been described with reference to the accompanying drawings, the example embodiments of the inventive concept are not limited to the above example embodiments, but can be implemented in various different forms. Those of ordinary skill in the art will understand that the example embodiments of the inventive concept can be practiced in other specific forms without changing the technical spirit or basic features of the inventive concept. Therefore, it should be understood that the above example embodiments are illustrative rather than restrictive in all respects.

[0092] So far, various example embodiments and effects of the inventive concept have been referred to Figures 1 to 9 The effects of the technical idea of the inventive concept are not limited to the effects described herein, and other unmentioned technical effects will be clearly understood by those of ordinary skill in the art to which the example embodiments of the inventive concept belong by referring to the claims given below.

[0093] Although the operations are shown in a particular order in the figures, it should be understood that desirable results may be obtained when the operations are performed in a different order and / or sequence, and / or when not all of the operations are performed. For example, in some cases, multitasking and parallel processing may have advantages. According to one or more example embodiments, it should be understood that the separation of various configurations is not necessary, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0094] Summarizing the detailed description, those of ordinary skill in the art will understand that many variations and modifications can be made to the various example embodiments without substantially departing from the principles of the inventive concept. Accordingly, the example embodiments of the present disclosure are used only in a general and descriptive sense and not for purposes of limitation. The scope of protection of the present disclosure should be construed in accordance with the appended claims, and all technical ideas within the scope of equivalents should be construed as being included within the scope of the claims of the present disclosure.

Claims

1. A method for detecting fixed pattern noise (FPN), the method comprising: Determine at least a portion of the image data received from an image sensor as a region of interest (ROI); Extract a plurality of pixel values from a first row of the ROI; Calculate a luminance value of the first row based on the extracted pixel values; Based on the calculated luminance value, Determine a first threshold related to a variance of the pixel values of the first row, and Determine a second threshold related to a difference between an average value of the pixel values of the first row and an average value of the pixel values of an adjacent row of the first row; Calculate a first score indicating the variance of the pixel values of the first row; Calculate a second score indicating a difference between an average value of the pixel values of the first row and an average value of the pixel values of the adjacent row of the first row; And Determine whether there is fixed pattern noise in the first row based on the first score and the first threshold and the second score and the second threshold.

2. The method according to claim 1, wherein A length of the ROI in a first direction is less than a length of the image data in the first direction; A length of the ROI in a second direction perpendicular to the first direction is equal to a length of the image data in the second direction; and The first direction is a row direction of a subset of the image data.

3. The method according to claim 1, wherein Calculating the first score further comprises: dividing the variance of the pixel values of the first row by the average value of the pixel values of the first row; and Calculating the second score further comprises: obtaining a difference between an average value of the pixel values of the first row and an average value of the pixel values of the adjacent row of the first row.

4. The method according to claim 1, wherein Calculating the second score further comprises: Calculating the second score for each of a second row adjacent to the first row upward and a third row adjacent to the first row downward.

5. The method according to claim 4, wherein In response to the first row being the first row of the image data, only calculate the second score of the third row; and In response to the first row being the last row of the image data, only calculate the second score of the second row.

6. The method according to claim 1, further comprising: In response to determining the presence of the fixed pattern noise; Display information indicating the presence of the fixed pattern noise; And Output the image data.

7. The method according to claim 1, wherein The first score and the second score are calculated for at least one pixel of the first row; In response to the image sensor using an RGB color filter, the at least one pixel is a green pixel; And In response to the image sensor using a CMYK color filter, the at least one pixel is a yellow pixel.

8. The method according to claim 1, further comprising: Rotate the image data by 90 degrees before setting at least a portion of the image data as the ROI.

9. An image signal processor, comprising: A processing circuit configured to: Process image data received from an image sensor; Detect whether there is fixed pattern noise in each row of the processed image data; Determine a part of the received processed image data as a region of interest (ROI); Extract the pixel values of each row of the ROI; Calculate the luminance value of each row of the ROI based on the extracted pixel values; Based on the calculated luminance values, Determine a first threshold related to the variance of the pixel values of each row of the ROI, and Determine a second threshold related to the difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of each row of the ROI; Calculate a first score indicating the variance of the pixel values of each row of the ROI; Calculate a second score indicating the difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of each row; And Determine whether there is fixed pattern noise in each row of the ROI based on the first score and the first threshold, and the second score and the second threshold.

10. The image signal processor according to claim 9, wherein, The processing circuit is further configured to: Perform at least one or any combination of color correction, automatic white balance, gamma correction, color saturation correction, bad pixel correction, and hue correction on the image data received from the image sensor.

11. The image signal processor according to claim 9, wherein The length of the ROI in the first direction is less than the length of the image data in the first direction; The length of the ROI in a second direction perpendicular to the first direction is equal to the length of the image data in the second direction; and The first direction is the row direction of the image data.

12. The image signal processor according to claim 9, wherein, The processing circuit is further configured to: Calculate the first score by dividing the variance of the pixel values of each row of the ROI by the average value of the pixel values of each row of the ROI; And Calculate the second score by obtaining the difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of each row.

13. The image signal processor according to claim 9, wherein The adjacent rows of each row of the ROI include: the row adjacent to each row of the ROI upward and the row adjacent to each row of the ROI downward; and The processing circuit is further configured to: In response to the row of the ROI being the first row of the image data, only calculate the second score of the row adjacent to the row downward; and In response to the row of the ROI being the last row of the image data, only calculate the second score of the row adjacent to the row upward.

14. The image signal processor according to claim 9, wherein, The processing circuit is further configured to: Store the result matrix in a memory, the result matrix being configured to store the detection result of the fixed pattern noise; In response to determining that there is the fixed pattern noise in each row of the ROI, record 1 in the corresponding row of the result matrix; And In response to determining that there is no fixed pattern noise in each row of the ROI, record 0 in the corresponding row of the result matrix.

15. The image signal processor according to claim 14, wherein, The processing circuit is further configured to: Display information related to the fixed pattern noise based on the result matrix.

16. The image signal processor according to claim 9, wherein, The processing circuit is further configured to: Calculate each of the first score and the second score using at least one pixel of each row of the ROI; In response to the image sensor using an RGB color filter, the at least one pixel is a green pixel; And In response to the image sensor using a CMYK color filter, the at least one pixel is a yellow pixel.

17. The image signal processor according to claim 9, wherein, The processing circuit is further configured to: Rotate the image data by 90 degrees.

18. An electronic device, comprising: An image sensor including a pixel array, the pixel array including a plurality of pixels, each pixel of the plurality of pixels including a first photodiode and a second photodiode, the light-receiving area of the second photodiode being larger than the light-receiving area of the first photodiode, wherein the pixel array is configured to: Output a first pixel signal based on a first conversion gain using the second photodiode within a first illuminance range; Output a second pixel signal based on a second conversion gain using the second photodiode within a second illuminance range; Output a third pixel signal based on the first conversion gain using the first photodiode within a third illuminance range; and Output a fourth pixel signal based on the second conversion gain using the first photodiode within a fourth illuminance range, wherein the first conversion gain is higher than the second conversion gain; The image sensor is configured to: Perform sampling on each of the first pixel signal to the fourth pixel signal; and Output image data based on the sampling result to a processing circuit; and The processing circuit is configured to: Determine a portion of the image data as a region of interest ROI; Extract the pixel values of each row of the ROI; Calculate the luminance value of each row of the ROI based on the extracted pixel values; Based on the calculated luminance values, Determine a first threshold related to the variance of the pixel values of each row of the ROI, and Determine a second threshold related to the difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of the ROI; Calculate a first score indicating the variance of the pixel values of each row of the ROI; Calculate a second score indicating the difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of the ROI; and Determine whether there is fixed pattern noise in each row of the ROI based on the first score and the first threshold and the second score and the second threshold.

19. The electronic device according to claim 18, wherein, The processing circuit is further configured to: Calculate the first score by dividing the variance of the pixel values of each row of the ROI by the average value of the pixel values of each row of the ROI; And Calculate the second score by obtaining the difference between the average value of the pixel values of each row of the ROI and the average value of the pixel values of the adjacent row of the ROI.

20. The electronic device according to claim 18, wherein, The processing circuit is further configured to: In response to the image data being generated based on the first pixel signal and the second pixel signal output within the first illuminance range and the second illuminance range respectively, calculate each of the first score and the second score using all pixels in each row of the ROI; In response to the image data being generated based on the third pixel signal and the fourth pixel signal output within the third illuminance range and the fourth illuminance range respectively, calculate each of the first score and the second score using at least one pixel in each row of the ROI; In response to the image sensor using an RGB color filter, the at least one pixel is a green pixel; And In response to the image sensor using a CMYK color filter, the at least one pixel is a yellow pixel.

Images