Image processing device including a neural network processor and method of operation

Abstract

An image processing device including a neural network processor and an operation method are disclosed. The image processing device includes: an image sensor including pixels generating first image data; and an image processing system. The image processing system includes: a neural network processor performing a cluster-level bad pixel correction operation on the first image data based on first coordinate information associated with a first type of bad pixel cluster to generate second image data, and a main processor performing a post-processing operation on the second image data to generate third image data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0082268 filed on Jul. 3, 2020, in the Korean Intellectual Property Office, the contents of which are incorporated herein by reference. Technical Field

[0002] Embodiments of the inventive concept relate to an image processing apparatus and an image processing system that perform an image processing operation using a neural network processor, and methods of operating the same. Background Art

[0003] Recently, as the demand for high-quality and high-resolution pictures and images increases, the size of pixels in image sensors is reduced as a greater number of pixels are accommodated. Therefore, due to certain process problems, the pixels of the image processing device may include bad pixels that appear in one or more arbitrary shapes and in one or more arbitrary positions.

[0004] Because a relatively large number of bad pixels having arbitrary shapes may not be used to generate a desired image, the performance of an image processing device may be degraded. Therefore, a technique for correcting pixel data associated with bad pixels is needed. However, bad pixels may occur in isolation or in clusters in unpredictable ways and times, and may result in variable performance. Therefore, a method of bad pixel correction is needed that takes into account both isolated bad pixels and clustered bad pixels. Summary of the invention

[0005] Embodiments of the inventive concept provide an image processing device, an image processing system, and a method of operating the image processing device and the image processing system, wherein the image processing device and the image processing system use a neural network processor to appropriately correct pixel data output from a pixel array including bad pixels (first type bad pixels or cluster type bad pixels, and / or second type bad pixels or isolated pixel type bad pixels).

[0006] According to one aspect of the inventive concept, an image processing device is provided, comprising: an image sensor including pixels generating first image data; and an image processing system. The image processing system comprises: a neural network processor configured to: perform a cluster-level bad pixel correction operation on the first image data based on first coordinate information associated with a first type of bad pixel cluster to generate second image data, and a main processor configured to perform a post-processing operation on the second image data to generate third image data.

[0007] According to one aspect of the inventive concept, an image processing system is provided, comprising: a preprocessor, receiving first four-in-one pixel (tetra) data and including a bad pixel module, the bad pixel module performing a pixel-level bad pixel correction operation on the first tetra data to generate pixel-level corrected first tetra data; a neural network processor, including a bad pixel network module, the bad pixel network module receiving the pixel-level corrected first tetra data, and performing a cluster-level bad pixel correction operation based on coordinate information to generate second tetra data; and a main processor, receiving the second tetra data, and comprising: a mosaic re-arrangement (remosaic) module, converting the second tetra data into corresponding data having a Bayer pattern; a demosaic (demosaic) module, converting the data having the Bayer pattern into corresponding data having a red-green-blue (RGB) pattern; a denoising module, removing noise from the data having the RGB pattern; and a sharpening module, performing a sharpening operation on the data having the RGB pattern.

[0008] According to one aspect of the inventive concept, a method for operating an image processing device is provided, the image processing device including an image sensor including pixels, a neural network processor and a main processor. The method includes: generating first image data using the pixels; performing a cluster-level bad pixel correction operation on the first image data using the neural network processor and based on coordinate information associated with a cluster including bad pixels among the pixels to generate second image data; and performing a post-processing operation on the second image data using the main processor to generate third image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram illustrating a neural network system according to an embodiment of the inventive concept;

[0011] Figure 2 is a conceptual diagram showing a possible neural network structure according to an embodiment of the inventive concept;

[0012] Figure 3 is a block diagram illustrating an image processing apparatus according to an embodiment of the inventive concept;

[0013] Figure 4A , Figure 4B and Figure 4C Yes and can be Figure 3 corresponding diagrams of exemplary pixel arrays corresponding to color filter arrays used in image processing devices;

[0014] Figure 5A and Figure 5Bis a corresponding conceptual diagram further illustrating the occurrence of bad pixels in a pixel array;

[0015] Fig. 6A , Figure 6B , Fig. 8A and Fig. 9 are respective block diagrams variously illustrating the operation of a neural network processor according to an embodiment of the inventive concept;

[0016] Fig. 7A and Figure 7B is a diagram further illustrating a cluster-level bad pixel correction operation according to an embodiment of the inventive concept;

[0017] Figure 8B is a flowchart illustrating an example operation of a neural network processor according to an embodiment of the inventive concept;

[0018] Fig.10 is a block diagram illustrating an example operation of a neural network processor according to an embodiment of the inventive concept; and

[0019] Fig.11 is a block diagram of an image processing apparatus according to an embodiment of the inventive concept. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the inventive concept will be described in some additional detail with reference to the accompanying drawings.

[0021] Figure 1 is a block diagram illustrating a neural network system 1 according to an embodiment of the inventive concept.

[0022] The neural network system 1 can train a neural network and use the (trained) neural network to analyze input data to determine specific information about the input data. In some embodiments, the neural network system 1 can determine information associated with a function, a condition, one or more component states, one or more component controls, etc. of an electronic device that includes the neural network system 1. In this regard, during one or more decision processes, the neural network system 1 can develop inferred information from the input data.

[0023] As examples of possible electronic devices, the neural network system 1 may be variously applied to, for example, smart phones, tablet devices, smart televisions (TVs), augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robots, medical devices, drones, advanced driver assistance systems (ADAS), image display devices, measuring devices, etc. The neural network system 1 may be variously applied to facilitate the execution (or operation) of, for example, speech recognition operations, image recognition operations, image classification operations, image processing operations, etc.

[0024] The neural network system 1 may be variously incorporated (e.g., mechanically mounted and / or electrically connected) by an electronic device. In this regard, the neural network system 1 may be internally disposed within an electronic device and / or externally associated with an electronic device. In some embodiments, the neural network system 1 may be an application processor (AP).

[0025] In the following, it will be assumed that the neural network system 1 has been suitably applied to an electronic device capable of performing digital image processing (and more specifically, a bad pixel correction operation for at least one cluster including bad pixels associated with a sensor module).

[0026] Reference Figure 1 , the neural network system 1 may include a sensor module 10, a neural network processor (or neural network device) 20, a central processing unit (CPU) 30, a random access memory (RAM) 40, a memory 50, and a bus 60 capable of variously connecting other components of the neural network system 1. As will be appreciated by those skilled in the art, the neural network system 1 may also include one or more input / output (I / O) modules, a security module, a power control module, and various additional processors. Here, each module or component in the neural network system 1 may be variously implemented in hardware, firmware, and / or software, wherein the corresponding software module may be executed using dedicated hardware components or general hardware components.

[0027] In some embodiments, some or all of the components of the neural network system 1 may be implemented on a single semiconductor chip. For example, the neural network system 1 may be implemented as a system on a chip (SoC). In some embodiments, the neural network system 1 may be referred to as an image chip.

[0028] The CPU 30 may be used to control the operation of the neural network system 1. Here, the CPU 30 may include a single core or multiple processor cores. The CPU 30 may be used to execute (or run) various operations and / or programs associated with programming (or writing), reading, erasing, and / or maintaining data about the RAM 40 and / or the memory 50.

[0029] The CPU 30 may execute an application program that controls the neural network processor 20 to perform various tasks associated with the neural network processor 20. In this regard, the neural network processor 20 may include at least one neural network model (such as, for example, a convolutional neural network (CNN), a region with a convolutional neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacked deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted Boltzmann machine (RBM), a full convolutional network, a long short-term memory (LSTM) network, a classification network, a plain residual network, a dense network, a hierarchical pyramid network, a full convolutional network, etc.) (potentially many different kinds of neural network models (such as, for example, a convolutional neural network (CNN), a region with a convolutional neural network (R-CNN), a region proposal network (RPN)) network, RPN), recurrent neural network (RNN), stacked deep neural network (S-DNN), state-space dynamic neural network (S-SDNN), deconvolution network, deep belief network (DBN), restricted Boltzmann machine (RBM), fully convolutional network, long short-term memory (LSTM) network, classification network, plain residual network, dense network, hierarchical pyramid network, fully convolutional network, etc.) at least one).

[0030] The neural network processor 20 may perform one or more neural network operations associated with input data stored in the RAM 40 and / or the memory 50 to generate a result information signal. The neural network processor 20 may be implemented as a neural network operation accelerator, a coprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), etc.

[0031] Assumptions Figure 1In an exemplary configuration, the sensor module 10 can be used to collect various information that is generally associated with the electronic device (internally or externally). For example, the sensor module 10 can sense or receive various sensor input signals (e.g., image signals or optical signals, audio signals, magnetic signals, biometric signals, touch signals, etc.) applied to or detected by the electronic device. The sensor input signal can then be converted into corresponding sensing data. To this end, the sensor module 10 may include one or more sensing devices (such as (for example), a microphone, an image capture device, a camera, an image sensor, a light detection and / or ranging (LIDAR) sensor, an ultrasonic sensor, an infrared (IR) sensor, a biosensor, a touch sensor, etc.).

[0032] In some embodiments, the sensory data may be provided as input data to the neural network processor 20. Alternatively or additionally, the input data may be stored in the memory 50 and thereafter provided to the neural network processor 20. In some embodiments, the neural network processor 20 may include a graphics processing unit (GPU) configured to process image data. Under this assumption, the sensory data may be processed by the GPU and provided to the memory 50 and / or the neural network processor 20.

[0033] Therefore, the sensor module 10 may include an image sensor capable of capturing images associated with the external environment surrounding the electronic device and generating image data. The generated image data provided by the sensor module 10 may be provided to the neural network processor 20 and / or stored in the memory 50 for subsequent processing by the neural network processor 20.

[0034] Expanding this example, the neural network processor 20 may receive image data from the sensor module 10 (or the memory 50) and perform a neural network operation based on the image data. In this regard, the neural network processor 20 may include a bad pixel (BP) network module 22 associated with a neural network operation based on one or more neural network models.

[0035] Therefore, the BP network module 22 may detect a specific bad pixel among a plurality of pixels (hereinafter collectively or partially referred to as "image sensor pixels") included in the image sensor of the sensor module 10 at the cluster level. The BP network module 22 may then correct the bad pixel data output by the cluster-level bad pixels. Hereinafter, the operation of correcting the bad pixel data output by one or more bad pixels will be referred to as a bad pixel correction operation. Reference will be made to Figure 6B An example of a cluster level bad pixel including a bad pixel is described with some additional detail.

[0036] In some embodiments, the BP network module 22 may receive first image data generated by the image sensor of the sensor module 10, and perform a cluster-level bad pixel correction operation on the first image data to generate cluster-level corrected first image data. The cluster-level bad pixel correction operation may be based on first coordinate information identifying one or more locations of bad pixel clusters (each bad pixel cluster includes at least two (2) first type bad pixels) among the image sensor pixels. In this manner, the cluster-level bad pixel correction operation may generate second image data from the first image data.

[0037] More specifically, the BP network module 22 may detect and select various image data associated with one or more regions of interest included in the first image data (hereinafter, "ROI data") based on the first coordinate information. The resulting "selected bad pixel data" may correspond to the image data associated with the ROI data, and the cluster-level bad pixel correction operation may be performed using the selected bad pixel data. Here, for example, the first coordinate information may include coordinates of pixel data respectively corresponding to the ROI data.

[0038] In some embodiments, the first coordinate information may be generated in whole or in part from masking information indicating the coordinates (or positions) of bad pixels in the image sensor pixels and derived, for example, during manufacturing / testing of the image sensor. That is, the neural network processor 20 may analyze the masking information and generate the first coordinate information, wherein the first coordinate information may be used to optimally perform cluster-level bad pixel correction. In some embodiments, the first coordinate information may be stored in the memory 50.

[0039] Therefore, in order to perform the cluster-level bad pixel correction operation, the BP network module 22 can access the memory 50 to read the first coordinate information. In addition, the first coordinate information can be periodically or dynamically updated based on, for example, the use cycle of the image sensor, one or more characteristics of one or more bad pixels, and / or other external factors.

[0040] In some embodiments, the BP network module 22 may perform a pixel-level bad pixel correction operation on the first image data received from the sensor module 10 based on second coordinate information associated with a second type of bad pixels (e.g., isolated bad pixels) among the image sensor pixels so as to generate pixel-level corrected first image data.

[0041] In this way, the BP network module 22 can perform a bad pixel correction operation including a cluster-level bad pixel correction operation based on the first coordinate information and / or a pixel-level bad pixel correction operation based on the second coordinate information. Here, the second coordinate information can be obtained from the mask information stored in the memory 50.

[0042] In some embodiments, the BP network module 22 may preferentially perform a pixel-level bad pixel correction operation on the first image data, and then perform a cluster-level bad pixel correction operation to generate the second image data. In another embodiment, the pixel-level bad pixel correction operation may be performed by another processor (e.g., CPU 30), and the BP network module 22 may perform a cluster-level bad pixel correction operation on the first image data in parallel with performing the pixel-level bad pixel correction operation on the first image data. Subsequently, the first image data on which the pixel-level bad pixel correction operation is performed may be merged with the first image data on which the cluster-level bad pixel correction operation is performed.

[0043] In some embodiments, once one or more bad pixel correction operations are performed, the neural network processor 20 may perform a reconstruction operation on the image data. In this regard, the reconstruction operation may be used to convert the format of the image data. For example, the reconstruction operation may convert image data having a tetra format (described below) to image data having a Bayer format, a red-green-blue (RGB) format, or a YUV format.

[0044] The neural network processor 20 may perform a reconstruction operation complementary to the reconstruction operation performed on the image data by the CPU 30. Alternatively, the neural network processor 20 may exclusively perform the reconstruction operation.

[0045] In some embodiments, in addition to the reconstruction operation, the neural network processor 20 may also perform one or more pre-processing operations and / or one or more post-processing operations on the image data. The neural network processor 20 may perform pre-processing operations or post-processing operations that are complementary to the pre-processing operations or post-processing operations performed on the image data by the CPU 30. Alternatively, the neural network processor 20 may exclusively perform one or more pre-processing operations and / or one or more post-processing operations. The neural network processor 20 may use the RAM 40 and / or the memory 50 to perform the above one or more operations.

[0046] exist Figure 1 In the example shown, the memory 50 may include volatile memory and / or non-volatile memory. The non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase-change random access memory (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), and / or a ferroelectric RAM (FRAM). The volatile memory may include a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous DRAM (SDRAM), a PRAM, an MRAM, an RRAM, and / or a FRAM.

[0047] In some embodiments, the memory 50 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF) card, a secure digital (SD) card, a micro SD card, a mini SD card, an extreme digital (xD) card, and a memory stick.

[0048] Figure 2 is a conceptual diagram showing, in one example, the structure of a neural network that may be used in an embodiment of the inventive concept. For example, Figure 2 The neural network can be applied to Figure 1 Neural network processor 20.

[0049] Reference Figure 2 , a neural network (NN) may include a plurality of layers (e.g., a first layer L1 to an nth layer Ln). Such a multi-layer neural network may be referred to as a deep neural network or a deep learning architecture. Each of the layers L1 to Ln may be a linear layer or a nonlinear layer. In some embodiments, a particular layer may include a combination of at least one linear layer and at least one nonlinear layer. For example, a linear layer may include a convolutional layer or a fully connected layer, and a nonlinear layer may include a pooling layer or an activation layer.

[0050] exist Figure 2 In the example shown in , the first layer L1 may include a convolutional layer, the second layer L2 may include a pooling layer, and the nth layer Ln may include, for example, a fully connected layer as an output layer. The neural network may also include an activation layer and / or one or more other layers configured to perform different types of operations.

[0051] Each of the first layer L1 to the nth layer Ln may (1) receive an input image frame or a feature map generated by a previous (or preceding) layer as an input feature map, (2) perform an operation on the input feature map, and (3) generate an output feature map or a recognition signal REC. Here, the feature map may represent data in which various features of the input data are represented. The first feature map to the nth feature map FM1, FM2, FM3, ... and FMn may have, for example, a two-dimensional (2D) matrix form or a three-dimensional (3D) matrix form (or referred to as a tensor) including a plurality of eigenvalues. The first feature map to the nth feature map FM1, FM2, FM3, ... and FMn may have a width W (or referred to as a column), a height H (or referred to as a row), and a depth D that may correspond to the x-axis, y-axis, and z-axis on the coordinates, respectively. In this case, the term depth D may represent the number of channels (e.g., CH).

[0052] The first layer L1 may convolve the first feature map FM1 with the weight map WM and generate a second feature map FM2. The weight map WM may have a 2D matrix form or a 3D matrix form including a plurality of weight values. The weight map WM may be referred to as a kernel. The weight map WM may filter the first feature map FM1 and is referred to as a filter or a kernel. The depth of the weight map WM (i.e., the number of channels) may be equal to the depth of the first feature map FM1 (i.e., the number of channels), and the channels of the weight map WM may be convolved with the channels of the first feature map FM1 corresponding thereto, respectively. The weight map WM may be traversed and shifted using the first feature map FM1 as a sliding window. During each shift, each weight value included in the weight map WM may be multiplied and added with all feature values ​​in an area where each weight value included in the weight map WM overlaps with the first feature map FM1. By convolving the first feature map FM1 with the weight map WM, one channel of the second feature map FM2 may be generated. Although in Figure 2 In FIG. 1 , one weight map WM is indicated, but in practice, multiple weight maps may be convolved with the first feature map FM1 to generate a second feature map FM2 of multiple channels. In other words, the number of channels of the second feature map FM2 may correspond to the number of weight maps.

[0053] The second layer L2 may use a pooling operation to change the spatial size of the second feature map FM2 and generate a third feature map FM3. The pooling operation may be referred to as a sampling operation or a downsampling operation. The 2D pooling window PW may be shifted on the second feature map FM2 in units of the size of the pooling window PW, and the maximum value (or average value) of the feature values ​​in the area overlapping with the pooling window PW may be selected. Therefore, the third feature map FM3 with a changed spatial size may be generated from the second feature map FM2. The number of channels of the third feature map FM3 may be equal to the number of channels of the second feature map FM2.

[0054] The n-th layer Ln may classify the category CL of the input data by combining the features of the n-th feature map FMn and generate a recognition signal REC corresponding to the category CL.

[0055] Figure 3 is a block diagram illustrating an image processing apparatus 1000 according to an embodiment of the inventive concept.

[0056] The image processing apparatus 1000 may be included or implemented as an electronic device configured to capture an image, display a captured image, and / or perform operations on a captured image. The image processing apparatus 1000 may be included or implemented as, for example, a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device may include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital audio camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MPEG-1 audio layer 3 (MP3) player, a handheld game console, an electronic book (e-book), or a wearable device. Optionally, the image processing apparatus 1000 may be included in various electronic devices associated with drones, ADAS, vehicles, furniture, manufacturing facilities, doors, measuring devices, and the like.

[0057] Reference Figure 3 The image processing device 1000 as a whole may include an image sensor 1100 and an image processing system 1200 .

[0058] The image processing apparatus 1000 may include various components such as a display, a user interface, etc.

[0059] Figure 3 The image processing system 1200 shown in FIG. 1 includes a preprocessor 100, a neural network processor 200, a main processor 300, and a memory 400. The preprocessor 100, the neural network processor 200, and the main processor 300 may be implemented as a single semiconductor chip or a group of semiconductor chips. For example, the image processing system 1200 and / or the image processing device 1000 may be implemented as a system on a chip (SoC). Although the preprocessor 100 and the main processor 300 are Figure 3 Although illustrated as separate components, the inventive concept is not limited thereto, and the pre-processor 100 and the main processor 300 may be implemented together in whole or in part.

[0060] The image sensor 1100 may include a color filter array (CFA) 1110 having a predetermined pattern. In this regard, the image sensor 1100 may be used to convert an optical signal associated with an object and incident to the optical lens LS into a corresponding electrical signal using the color filter array 1110 to generate first image data (IDATA).

[0061] The color filter array 1110 may be implemented to support one or more patterns (such as one or more Bayer patterns, one or more tetra patterns, etc.). For example, the color filter array 1110 may have a pattern corresponding to a four-in-one pixel unit (tetra cell, or referred to as a four-pixel-in-one unit, a pixel four-in-one unit) or a pattern corresponding to a nine-in-one pixel unit (nonacell, or referred to as a nine-pixel-in-one unit, a pixel nine-in-one unit). In the following, it will be assumed that the color filter array 1110 corresponds to a tetra pattern. However, the inventive concept is not limited thereto, and the color filter array 1110 may correspond to a pattern corresponding to other multi-in-one pixel units (or referred to as multi-pixel-in-one units, pixel multi-in-one units).

[0062] The image sensor 1100 may include, for example, a pixel array of arranged (e.g., two-dimensionally arranged) pixels and a readout circuit. The pixel array may be used to convert one or more incident light signals into corresponding one or more electrical signals. The pixel array may be implemented as a photoelectric conversion device (such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) device, or various other photoelectric conversion devices). The readout circuit may generate raw data based on the electrical signal provided from the pixel array, and output the raw data from which noise may be removed as first image data. The image sensor 1100 may be implemented as a semiconductor chip or package including a pixel array and a readout circuit.

[0063] In some embodiments, the pre-processor 100 may be used to perform at least one pre-processing operation on the first image data (such as a crosstalk (X-talk) correction operation, a pixel-level bad pixel correction operation, etc.).

[0064] exist Figure 3 In the illustrated example, it is assumed that the neural network processor 200 includes a BP network module 220, wherein the BP network module 220 can be used to perform a cluster-level bad pixel correction operation on the first image data. As an example, the BP network module 220 can (1) read the first coordinate information from the memory 400, (2) select ROI data from the first image data IDATA based on the first coordinate information, (3) detect bad pixel data corresponding to a cluster including bad pixels among the ROI data, and (4) perform a cluster-level bad pixel correction operation.

[0065] With this configuration, for example, the BP network module 220 may perform a cluster-level bad pixel correction operation in parallel with a pixel-level bad pixel correction operation performed by the preprocessor 100. Thereafter, the main processor 300 may be used to merge the first image data on which the preprocessor 100 has performed the pixel-level bad pixel correction operation and the first image data on which the neural network processor 200 has performed the cluster-level bad pixel correction operation. Then, one or more post-processing operations may be performed on the merged first image data.

[0066] In some embodiments, the neural network processor 200 may perform all or some of the preprocessing operations performed by the preprocessor 100 without using the preprocessor 100. For example, the neural network processor 200 may perform an X-talk correction operation and a pixel-level bad pixel correction operation on the first image data.

[0067] The image sensor pixels included in the image sensor 1100 may include static bad pixels and / or dynamic bad pixels. Here, the static bad pixels may continuously output bad pixel data regardless of environmental factors and noise. In contrast, the dynamic bad pixels may output bad pixel data or normal pixel data according to various factors (such as environmental factors and / or noise).

[0068] Therefore, the BP network module 220 can detect bad pixel data in the first image data generated by static bad pixels based on the masking information, and correct the bad pixel data. Alternatively or additionally, the BP network module 220 can detect bad pixel data in the first image data generated by dynamic bad pixels based on the masking information and the pattern of each pixel data value of the first image data, and correct the detected bad pixel data. The BP network module 220 can then perform a cluster-level bad pixel correction operation on the first image data on which the pixel-level bad pixel correction operation has been performed.

[0069] The main processor 300 may perform post-processing operations including a remosaic operation and a demosaic operation on the second image data output by the neural network processor 200, and may also be used to convert (or change) the format of the image data. For example, the main processor 300 may convert the second image data from a tetra format to a Bayer format (or an RGB format) in order to generate third image data (e.g., full-color image data).

[0070] In some embodiments, the neural network processor 200 may perform all or some of the pre-processing operations performed by the main processor 300 without using the main processor 300. For example, the neural network processor 200 may perform mosaic rearrangement on the second image data in the tetra format to convert the second image data from the tetra format to the Bayer format. Alternatively, the neural network processor 200 may be used to demosaic the second image data in the Bayer format to generate the second image data in the RGB format.

[0071] Will refer to FIG. 6A to FIG. 10 Exemplary operation of the neural network processor 200 according to various embodiments is described with some additional detail.

[0072] According to an embodiment of the inventive concept, Figure 3The image processing apparatus 1000 may perform a cluster-level bad pixel correction operation on bad pixel data generated by various clusters using the neural network processor 200. In addition, the image processing apparatus 1000 may perform one or more pre-processing operations and / or one or more post-processing operations suitable for the cluster-level bad pixel correction operation using the neural network processor 200. As a result, the image processing apparatus 1000 may generate a high-resolution and high-quality image.

[0073] Figure 4A , Figure 4B and Figure 4C is a diagram showing an embodiment according to the inventive concept that may include Figure 3 A conceptual diagram of various arrangements of a pixel array (PX_Array) in a color filter array 1110.

[0074] Reference Figure 4A , the pixel array may include pixels arranged in a matrix of rows and columns. Here, a shared pixel may be defined as a unit including pixels arranged in two rows and two columns and including four (4) sub-pixels. Therefore, each shared pixel may include four (4) photodiodes corresponding to the four (4) sub-pixels, respectively. Therefore, the pixel array may include a first shared pixel SP0 to a sixteenth shared pixel SP15.

[0075] Figure 4A The pixel array (PX_Array) may further include a color filter so that the first to sixteenth shared pixels SP0 to SP15 can sense various colors. As an example, the color filter may include filters configured to sense red (R), green (G), and blue (B), respectively.

[0076] Each of the first shared pixel SP0 to the sixteenth shared pixel SP15 may include a sub-pixel in which the same color filter is placed. For example, each of the first shared pixel SP0, the third shared pixel SP2, the ninth shared pixel SP8, and the eleventh shared pixel SP10 may include a sub-pixel including a blue (B) color filter. Each of the second shared pixel SP1, the fourth shared pixel SP3, the fifth shared pixel SP4, the seventh shared pixel SP6, the tenth shared pixel SP9, the twelfth shared pixel SP11, the thirteenth shared pixel SP12, and the fifteenth shared pixel SP14 may include a sub-pixel including a green (G) color filter. Each of the sixth shared pixel SP5, the eighth shared pixel SP7, the fourteenth shared pixel SP13, and the sixteenth shared pixel SP15 may include a sub-pixel including a red (R) color filter.

[0077] In addition, each of a group including a first shared pixel SP0, a second shared pixel SP1, a fifth shared pixel SP4 and a sixth shared pixel SP5, a group including a third shared pixel SP2, a fourth shared pixel SP3, a seventh shared pixel SP6 and an eighth shared pixel SP7, a group including a ninth shared pixel SP8, a tenth shared pixel SP9, a thirteenth shared pixel SP12 and a fourteenth shared pixel SP13, and a group including an eleventh shared pixel SP10, a twelfth shared pixel SP11, a fifteenth shared pixel SP14 and a sixteenth shared pixel SP15 can be arranged in a pixel array to correspond to a Bayer pattern.

[0078] However, it should be noted at this time that Figure 4A The pixel array (PX_Array) is only one example of one or more pixel arrays and / or one or more color filters that can be used with respect to embodiments of the inventive concept. For example, the pixel array may include various types of color filters (such as filters configured to sense yellow, cyan, magenta, and green, respectively). Alternatively, the color filter may include filters configured to sense red, green, blue, and white, respectively. In addition, the pixel array may include a greater number of shared pixels than the number of shared pixels described above, and the shared pixels (e.g., SP0 to SP15) that are constituted may be arranged differently.

[0079] exist Figure 4B In the illustrated embodiment, each of the first shared pixel SP0, the second shared pixel SP1, the fifth shared pixel SP4, and the sixth shared pixel SP5 may include nine (9) sub-pixels. The first shared pixel SP0 may include nine sub-pixels including a blue (B) color filter. Each of the second shared pixel SP1 and the fifth shared pixel SP4 may include nine sub-pixels including a green (G) color filter. The sixth shared pixel SP5 may include nine sub-pixels including a red (R) color filter. This particular arrangement of the first shared pixel SP0, the second shared pixel SP1, the fifth shared pixel SP4, and the sixth shared pixel SP5 may be referred to as a tetra unit.

[0080] exist Figure 4C In the illustrated embodiment, each of the first shared pixel SP0, the second shared pixel SP1, the fifth shared pixel SP4, and the sixth shared pixel SP5 may include sixteen (16) sub-pixels. The first shared pixel SP0 may include sixteen sub-pixels including a blue (B) color filter. Each of the second shared pixel SP1 and the fifth shared pixel SP4 may include sixteen sub-pixels including a green (G) color filter. The sixth shared pixel SP5 may include sixteen sub-pixels including a red (R) color filter. This particular arrangement of the first shared pixel SP0, the second shared pixel SP1, the fifth shared pixel SP4, and the sixth shared pixel SP5 may be referred to as a nine-in-one pixel unit.

[0081] Figure 5A (include Figure 5A (a), (b) and (c)) and Figure 5B (include Figure 5B (a), (b), (c), (d), and (e)) are respective conceptual diagrams illustrating the existence of a specific bad pixel (BP) included in a pixel array.

[0082] Figure 5A (a) shows an example of a pixel array including one (1) bad pixel BP; Figure 5A (b) shows an example where the pixel array includes two (2) bad pixels BP that are isolated from each other; and Figure 5A (c) shows an example where the pixel array includes five (5) bad pixels BP that are considered to be isolated from each other.

[0083] Therefore, if Figure 5A (a) to Figure 5A (b) to Figure 5A As suggested by the progression of (c), an increasing number of bad pixels BP that are discontinuous with each other may be generated. In this case, a pixel-level bad pixel correction operation may be suitable for correcting the bad pixels BP. In addition, Figure 5A The bad pixels BP shown in FIG. 4 may be one or more static bad pixels or one or more dynamic bad pixels.

[0084] In contrast, Figure 5B (a), (b), (c), (d) and (e) show examples in which the bad pixels BP are continuous with each other and can form bad pixel clusters of arbitrary shapes. Figure 5B In (a), a single shared pixel SP may include four (4) bad pixels, thereby forming a cluster. Figure 5B In (b), the adjacent shared pixels SP may include sixteen (16) bad pixels BP arranged continuously relative to each other, thereby forming a cluster. Figure 5B In (c), different shared pixels SP may include twenty-five (25) bad pixels BP that are consecutive to each other, thereby forming a plurality of clusters. Figure 5B In (d), ten (10) bad pixels BP arranged in parallel in any axial direction may form a cluster having a line pattern (eg, a line-type cluster) on the pixel array. Figure 5B In (e), a large number of bad pixels BP may be arranged in so-called scratch-type clusters on the pixel array.

[0085] Therefore, if Figure 5B As shown in (a), (b), (c), (d) and (e) of FIG. 5 , a plurality of bad pixels BP may be continuous with each other in various ways to form different types of clusters. In this case, a cluster-level bad pixel correction operation may be suitable for correcting the bad pixels BP.

[0086] Fig. 6A and Figure 6B 1 is a block diagram further illustrating possible operations of the corresponding image processing systems 1200a and 1200b according to an embodiment of the inventive concept. Fig. 6A and Figure 6B For example, the image processing systems 1200a and 1200b may further include additional modules configured to provide one or more pre-processing operations and / or one or more post-processing operations.

[0087] Reference Fig. 6A , the image processing system 1200a may include a preprocessor 100a, a neural network processor 200a, and a main processor 300a. The preprocessor 100a may include an X-talk correction module 120a and a bad pixel (BP) correction module 130a. The neural network processor 200a may include a BP network module 220a. The main processor 300a may include a mosaic rearrangement module 310a, a demosaicing module 320a, a denoising module 330a, and a sharpening module 340a.

[0088] The preprocessor 100a may receive the first image data (IDATAa) having a tetra format (hereinafter, "first tetra data") (or first all-in-one pixel data having an all-in-one pixel format), and perform a preprocessing operation including an X-talk correction operation and a pixel-level bad pixel correction operation on the first tetra data (or the first all-in-one pixel data). That is, the BP correction module 130a may perform a pixel-level bad pixel correction operation based on the masking information BP_MI. The masking information BP_MI may include: Figure 5A In addition, the masking information BP_MI may include the following: Figure 5B The diagram includes location information of bad pixels in one or more clusters.

[0089] Then, the BP network module 220a may (1) receive the first tetra data after pixel-level correction (or the first all-in-one pixel data after pixel-level correction) from the preprocessor 100a, (2) perform a cluster-level bad pixel correction operation on the tetra data after pixel-level correction based on the coordinate information BP_CI, so as to (3) generate the second tetra data (or the second all-in-one pixel data) (IDATAb). That is, in some embodiments, the BP network module 220a may (1) select the ROI data included in the first tetra data, (2) detect the pixel data corresponding to the cluster from the ROI data, and (3) perform the cluster-level bad pixel correction operation. Here, the coordinate information BP_CI may include the coordinates of the pixel data respectively including at least some of the ROI data. The coordinate information BP_CI may be generated based on the mask information BP_MI.

[0090] In some embodiments, the neural network processor 200a may (1) extract the position information of the bad pixel included in the mask information BP_MI using the coordinates of the bad pixel, (2) select the best ROI data for the cluster-level bad pixel correction operation, and (3) generate the coordinate information BP_CI. The neural network processor 200a may obtain the mask information BP_MI during the operation of the production image processing system 1200a to generate the coordinate information BP_CI, and store the coordinate information BP_CI in the non-volatile memory of the image processing system 1200a. In some embodiments, the BP network module 220a may perform a white balance operation on the first tetra data before performing the cluster-level bad pixel correction operation.

[0091] The main processor 300a may receive the second tetra data (or the second all-in-one pixel data) (IDATAb) from the neural network processor 200a. The mosaic rearrangement module 310a may convert the second tetra data (or the second all-in-one pixel data) into corresponding data having a Bayer pattern, and the demosaicing module 320a may then convert the data having the Bayer pattern into data (IDATAc) having an RGB pattern (e.g., full-color image data). Subsequently, the denoising module 330a may perform an operation of removing noise from the RGB data, and the sharpening module 340a may perform a sharpening operation on the RGB data. Here, the order of the denoising operation and the sharpening operation may be changed. The main processor 300a may perform one or more post-processing operations described above with respect to the second tetra data (IDATAb) and / or the RGB data (IDATAc).

[0092] Reference Figure 6B The image processing system 1200b may include a preprocessor 100b, a neural network processor 200b, an image merging unit 240b and a main processor 300b. Fig. 6A The neural network processor 200a and Figure 6B The differences between the neural network processor 200b.

[0093] The neural network processor 200b may directly receive the first tetra data (IDATAa) from the X-talk correction module 120b. The BP network module 220b may operate in parallel with the BP correction module 130b. That is, the BP network module 220b may perform a cluster-level bad pixel correction operation on the first tetra data. In parallel with the cluster-level bad pixel correction operation, the BP correction module 130b may perform a pixel-level bad pixel correction operation on the first tetra data. Then, the image merging unit 240b may merge the data output by the BP correction module 130b with the data output by the BP network module 220b to generate the second tetra data (IDATAb).

[0094] Fig. 7A is a flow chart further illustrating a cluster-level bad pixel correction operation according to an embodiment of the inventive concept, Figure 7B is a conceptual diagram showing one possible structure of a data block (IDATA).

[0095] Reference Fig. 7A , the neural network processor may receive the image data and the coordinate information of the cluster (each of the image data and the cluster may include one or more bad pixels) (S100). Thereafter, the neural network processor may select ROI data from the image data based on the coordinate information, detect pixel data corresponding to the cluster from the ROI data, and perform a bad pixel correction operation on the detected pixel data (S110) so as to output the corrected image data (IDATA) (S120).

[0096] Further references Figure 7B , the image data (IDATA) may include ROI data (e.g., first ROI data ROI_1 to fourth ROI data ROI_4). Here, the first ROI data ROI_1 to the fourth ROI data ROI_4 may be sequentially selected as the target ROI data T_ROI, and thus, the pixel data corresponding to the clusters (each cluster including one or more bad pixels) may be detected. The coordinate information may include first coordinates (X1, Y1) to fourth coordinates (X4, Y4) indicating the pixel data of the first ROI data ROI_1 to the fourth ROI data ROI_4, respectively. For example, the neural network processor may use the first coordinates (X1, Y1) to detect the first ROI data ROI_1. The ROI data may be of different sizes. However, in some embodiments, the respective ROI data may have the same size. When the respective ROI data have different sizes, the coordinate information may also include size information of the ROI data.

[0097] Therefore, in some embodiments, a neural network processor consistent with embodiments of the inventive concept may perform a cluster-level bad pixel correction operation by optionally using selected ROI data based on coordinate information to achieve fast image processing operations.

[0098] Fig. 8A and Figure 8B 2 is a diagram for describing possible operations of a neural network processor 200 c in an image processing system 1200 c according to an embodiment of the inventive concept.

[0099] Reference Fig. 8A The image processing system 1200c may include a preprocessor 100c, a neural network processor 200c and a main processor 300c. Fig. 6A The neural network processor 200a and Fig. 8A The differences between the neural network processor 200c.

[0100] In some embodiments, the neural network processor 200c may include a BP correction module 210c and a BP network module 220c. The BP correction module 210c may receive the first tetra data (IDATAa) from the X-talk correction module 120c of the preprocessor 100c, and perform a pixel-level bad pixel correction operation based on the masking information BP_MI. Subsequently, the BP network module 220c may perform a cluster-level bad pixel correction operation on the first tetra data received from the BP correction module 210c to generate a second tetra data (IDATAb).

[0101] Reference Figure 8B , the BP correction module 210c may detect at least one static bad pixel based on the masking information BP_MI (e.g., the second coordinate information), and correct the pixel data corresponding to the at least one static bad pixel (S200). The BP correction module 210c may detect at least one dynamic bad pixel based on the masking information BP_MI (e.g., the second coordinate information) and / or the correction result for at least one static bad pixel, and correct the pixel data corresponding to the at least one dynamic bad pixel (S210). After the correction of at least one static bad pixel and the correction of at least one dynamic bad pixel (S200 and S210), the BP network module 220c may perform a cluster-level bad pixel correction operation on the image data (S220). However, the inventive concept is not limited to this method, and the cluster-level bad pixel correction operation on the image data may be followed by a pixel-level bad pixel correction operation.

[0102] Fig. 9 and Fig.10 is a block diagram further illustrating possible operations of the neural network processors 200d and 200e according to an embodiment of the inventive concept.

[0103] Reference Fig. 9 The image processing system 1200d may include a preprocessor 100d, a neural network processor 200d and a main processor 300d. Fig. 6A The neural network processor 200a and Fig. 9 The difference between the neural network processor 200d.

[0104] The neural network processor 200d may include a BP network module 220d and a mosaic rearrangement module 230d. The mosaic rearrangement module 230d may perform mosaic rearrangement on the tetra data output by the BP network module 220d and convert the tetra data into data (IDATAb) having a Bayer format. The mosaic rearrangement module 230d may output the data having a Bayer format to the main processor 300d. The main processor 300d may include a demosaicing module 320d, a denoising module 330d, and a sharpening module 340d. Therefore, the main processor 300d may post-process the data having a Bayer format so as to generate RGB data (IDATAc).

[0105] Reference Fig.10 , the image processing system 1200e may include a neural network processor 200e. Fig. 6A In contrast, the neural network processor 200e can perform a pre-processing operation, a pixel-level bad pixel correction operation, a cluster-level bad pixel correction operation, and a post-processing operation on the first tetra data (IDATAa) to generate RGB data (IDATAc). That is, the neural network processor 200e can be used to perform the following operations: Fig. 6A All operations described for the pre-processor 100a and the main processor 300a.

[0106] Fig.11 is a block diagram illustrating an image processing apparatus 2000 according to an embodiment of the inventive concept. Here, it is assumed that the image processing apparatus 2000 is a portable terminal.

[0107] Reference Fig.11 , the image processing apparatus 2000 may include an application processor (AP) 2100, an image sensor 2200, a display device 2900, a working memory 2500, a storage device 2600, a user interface 2700, and a wireless transceiver unit 2800. The AP 2100 may include an image signal processor (ISP) 2300 and a neural network processor (or NNP) 2400. Figure 1 The image processing method described in the embodiments of the present invention may be applied to the ISP 2300 and the NNP 2400. In some embodiments, the ISP 2300 and the NNP 2400 may be implemented as an integrated circuit (IC) separate from the AP 2100.

[0108] The AP 2100 may control operations of the image processing apparatus 2000 , and may be implemented as a system on chip (SoC) configured to drive one or more application programs and / or an operating system.

[0109] In this regard, the AP 2100 may control the operation of the ISP 2300 and provide the converted image data generated by the ISP 2300 to the display device 2900 and / or store the converted image data in the storage device 2600 .

[0110] The image sensor 2200 may generate image data (eg, raw image data) based on the received optical signal and provide the image data to the ISP 2300 .

[0111] The working memory 2500 may be implemented as a volatile memory (eg, DRAM and SRAM) or a nonvolatile memory (eg, FeRAM, RRAM, and PRAM). The working memory 2500 may store programs and / or data processed or executed by the AP 2100.

[0112] The storage device 2600 may be implemented as a non-volatile memory device, such as a NAND flash memory device and a resistive memory. For example, the storage device 2600 may be provided as a memory card (e.g., a multimedia card (MMC), an embedded MMC (eMMC), a secure digital (SD) card, and a micro SD card). The storage device 2600 may store data and / or programs for executing algorithms for controlling image processing operations of the ISP 2300. The data and / or programs may be loaded into the working memory 2500 during image processing operations. In one embodiment, the storage device 2600 may store image data generated by the ISP 2300 (e.g., converted image data or post-processed image data).

[0113] The user interface 2700 may be implemented as one of various devices capable of receiving a user's input, such as a keyboard, a button key panel, a touch panel, a fingerprint sensor, and a microphone. The user interface 2700 may receive the user's input and provide a signal corresponding to the user's input to the AP 2100.

[0114] The wireless transceiver unit 2800 may include a transceiver 2810 , a modem 2820 , and an antenna 2830 .

[0115] While the inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the scope of the appended claims.

Claims

1. An image processing device, comprising: an image sensor comprising pixels that generate first image data; as well as Image processing system, including: a neural network processor configured to: perform a cluster-level bad pixel correction operation on the first image data based on first coordinate information associated with a first type of bad pixel cluster to generate second image data; and a main processor configured to perform a post-processing operation on the second image data to generate third image data, The neural network processor is configured to extract the position information of the first type of bad pixel cluster included in the masking information derived during the manufacturing / testing of the image sensor, and select the best region of interest data for the cluster-level bad pixel correction operation to generate first coordinate information, and store the first coordinate information in the non-volatile memory of the image processing system, The neural network processor is configured to: select region of interest data included in the first image data based on first coordinate information stored in the non-volatile memory, and detect pixel data corresponding to the first type of bad pixel cluster from the region of interest data.

2. The image processing apparatus according to claim 1, wherein: The first coordinate information includes coordinates of pixel data respectively indicating the region of interest data.

3. The image processing device according to claim 1, further comprising: The non-volatile memory is configured to store first coordinate information.

4. The image processing device according to claim 1, further comprising: The preprocessor is configured to perform a pixel-level bad pixel correction operation on the first image data based on second coordinate information associated with the second type of bad pixels to generate pixel-level corrected first image data.

5. The image processing apparatus according to claim 4, wherein: The neural network processor is further configured to perform a cluster-level bad pixel correction operation on the first image data after pixel-level correction to generate second image data.

6. The image processing apparatus according to claim 4, wherein: The neural network processor is further configured to: perform a cluster-level bad pixel correction operation in parallel with the pixel-level bad pixel correction operation performed by the preprocessor to generate cluster-level corrected first image data, and The main processor is further configured to merge the pixel-level corrected first image data with the cluster-level corrected first image data to generate third image data.

7. The image processing apparatus according to claim 1, wherein: The neural network processor is further configured to: perform a pixel-level bad pixel correction operation on the first image data based on second coordinate information associated with the second type of bad pixels, and thereafter, A cluster-level bad pixel correction operation is performed on the first image data after pixel-level correction by a neural network processor.

8. The image processing apparatus according to claim 7, wherein: The pixel includes at least one static bad pixel and at least one dynamic bad pixel, and The neural network processor is also configured to: detect pixel data corresponding to the at least one static bad pixel and the at least one dynamic bad pixel from the first image data by referring to the second coordinate information to generate detected pixel data, and correct the detected pixel data.

9. The image processing apparatus according to any one of claims 1 to 8, wherein: The pixel includes at least one of a four-in-one pixel unit and a nine-in-one pixel unit.

10. The image processing apparatus according to any one of claims 1 to 8, wherein: The neural network processor is further configured to perform at least one of a crosstalk correction operation on the first image data and a mosaic rearrangement operation on the second image data.

11. The image processing apparatus according to any one of claims 1 to 8, wherein: The neural network processor is also configured to perform a white balancing operation on the first image data before performing the cluster-level bad pixel correction operation.

12. The image processing apparatus according to any one of claims 1 to 8, wherein: The first image data and the second image data are four-in-one pixel data respectively, and The third image data is red-green-blue data.

13. The image processing apparatus according to any one of claims 1 to 8, wherein: At least one of a general residual network, a dense network, a hierarchical pyramid network, and a fully convolutional network is applied to a neural network processor.

14. The image processing apparatus according to any one of claims 1 to 8, wherein: The third image data is full-color image data, and The main processor is further configured to perform a mosaic rearrangement operation on the second image data to generate image data having a second pattern, and to perform a demosaicing operation on the image data having the second pattern to generate full-color image data.

15. The image processing apparatus according to any one of claims 1 to 8, wherein: The first type bad pixel cluster includes at least one of a scratch type cluster and a line type cluster.

16. An image processing system, comprising: a preprocessor that receives first all-in-one pixel data generated using pixels of the image sensor and includes a bad pixel module that performs a pixel-level bad pixel correction operation on the first all-in-one pixel data to generate pixel-level corrected first all-in-one pixel data; A neural network processor, comprising a bad pixel network module, the bad pixel network module receiving the first all-in-one pixel data after pixel-level correction, and performing a cluster-level bad pixel correction operation based on the coordinate information to generate second all-in-one pixel data; as well as A main processor receives the second all-in-one pixel data and includes: A mosaic rearrangement module, converting the second all-in-one pixel data into corresponding data having a Bayer pattern; a demosaicing module that converts data having a Bayer pattern into corresponding data having a red-green-blue pattern; a denoising module to remove noise from corresponding data having a red-green-blue pattern; and A sharpening module performs a sharpening operation on the corresponding data having a red-green-blue pattern, wherein the neural network processor is configured to: extract location information of the first type of bad pixel cluster included in the masking information derived during the manufacture / testing of the image sensor, and select the best region of interest data for the cluster-level bad pixel correction operation to generate coordinate information, and store the coordinate information in the non-volatile memory of the image processing system, The neural network processor is configured to: select region of interest data included in the first all-in-one pixel data after pixel-level correction based on coordinate information stored in a non-volatile memory, and detect pixel data corresponding to a first type of bad pixel cluster from the region of interest data.

17. The image processing system according to claim 16, wherein: The preprocessor further includes a crosstalk correction module that performs a crosstalk correction operation on the first all-in-one pixel data before the pixel-level bad pixel correction operation is performed, and The pixel level bad pixel correction operation is based on masking information including location information indicative of the second type of bad pixels.

18. A method of operating an image processing device, the image processing device comprising an image sensor including pixels and an image processing system, the image processing system comprising a neural network processor and a main processor, the method comprising: generating first image data using pixels; performing a cluster-level bad pixel correction operation on the first image data using a neural network processor and based on coordinate information associated with a cluster including a bad pixel among the pixels to generate second image data; as well as performing a post-processing operation on the second image data to generate third image data using a main processor, The method further includes extracting location information of clusters including bad pixels included in masking information derived during manufacturing / testing of the image sensor, and selecting optimal region of interest data for cluster-level bad pixel correction operation to generate coordinate information, and storing the coordinate information in a non-volatile memory of the image processing system, The step of performing cluster-level bad pixel correction operation includes: selecting region of interest data included in the first image data based on coordinate information stored in the nonvolatile memory, and detecting pixel data corresponding to a cluster including bad pixels from the region of interest data.

Images