Image sensing device and operating method thereof

Abstract

Provided herein may be an image sensing device and an operation method thereof. The image sensing device may include: an image sensor configured to acquire an image including a plurality of pixel values; a memory configured to store a reference gain value for each of a plurality of block regions included in the image; and an image processor configured to calculate a gain value included in each of the plurality of block regions using the reference gain value, and output a corrected image in which the reference gain value and the gain value are applied to the plurality of pixel values, wherein the block regions include a first block region and a second block region, the second block region being shorter than the first block region from the center of the image and having a size larger than the size of the first block region.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate generally to electronic devices, and more particularly, to image sensing devices and methods of operating the same. Background Art

[0002] An image-sensing (image sensing) device is a device that acquires an image. Recently, with the development of the computer industry and the communication industry, the demand for image sensing devices has increased in various electronic devices such as smart phones, digital cameras, game consoles, Internet of Things (IoT) devices, robots, security cameras, medical cameras, and autonomous vehicles.

[0003] The image sensing device may correct the image to improve the image quality. The reason for this is that the quality of the image may be reduced due to the pixel structure of the image sensing device, the wavelength characteristics of light, the manufacturing error of the lens, the alignment error between the lens and the image sensor, etc. Specifically, when the imbalance between the green channel (e.g., channel Gb and channel Gr) corresponding to one of the colors to which human vision is most sensitive increases, factors that reduce the image quality, such as grid noise, may appear in the image.

[0004] The image sensing device may allow data to be used for correction to be pre-stored in a memory. However, since resources such as hardware resources are limited, an optimization task is required to minimize the resources required for data processing and storage while maintaining performance such as improved image quality. Summary of the invention

[0005] Various embodiments of the present disclosure relate to an image sensing device and a method of operating the image sensing device that can reduce the amount of data to be used for image correction and a calculation load.

[0006] An embodiment of the present disclosure may provide an image sensing device. The image sensing device may include: an image sensor configured to acquire an image including a plurality of pixel values; a memory configured to store a reference gain value for each of a plurality of block regions included in the image; and an image processor configured to calculate a gain value included in each of the plurality of block regions using the reference gain value, and output a corrected image in which the reference gain value and the gain value are applied to the plurality of pixel values, wherein the plurality of block regions include a first block region and a second block region, the second block region being shorter than the first block region from the center of the image and having a size larger than the size of the first block region.

[0007] An embodiment of the present disclosure may provide a method for operating an image sensing device. The method may include: calculating a reference gain value for each of a plurality of block regions included in a corrected image acquired by the image sensing device; calculating a gain value included in each of the plurality of block regions using the reference gain value; storing the reference gain value and the gain value in a memory; and when an image is acquired by the image sensing device, outputting a corrected image in which the reference gain value and the gain value stored in the memory are applied to a plurality of pixel values, wherein the plurality of block regions include a first block region and a second block region, the second block region being shorter than the first block region from the center of the image and having a size larger than the size of the first block region. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram illustrating an image sensing device according to an embodiment of the present disclosure.

[0009] Figure 2 is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0010] Figure 3A is a diagram illustrating a pixel array according to an embodiment of the present disclosure.

[0011] Figure 3B is a diagram illustrating an image acquired by an image sensor according to an embodiment of the present disclosure.

[0012] Figure 4 is a diagram illustrating an image processor according to an embodiment of the present disclosure.

[0013] Figure 5 is a diagram illustrating a calibration image according to an embodiment of the present disclosure.

[0014] Figure 6 is a diagram illustrating a gain graph according to an embodiment of the present disclosure.

[0015] Figure 7 is a diagram illustrating any one block region according to an embodiment of the present disclosure.

[0016] FIG. 8A to FIG. 8C is a diagram illustrating a method of calculating a reference gain value according to an embodiment of the present disclosure.

[0017] Fig.9A and Fig. 9B is a diagram illustrating a method of calculating a gain value according to an embodiment of the present disclosure.

[0018] Fig.10 is a diagram illustrating a corrected image according to an embodiment of the present disclosure.

[0019] Fig.11is a flowchart illustrating a method of operating an image sensing device according to an embodiment of the present disclosure.

[0020] Fig.12 is a flowchart illustrating a method of operating an image sensing device according to an embodiment of the present disclosure.

[0021] Fig.13 is a diagram illustrating a computing system including an image sensing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The specific structural or functional descriptions in the embodiments of the present disclosure introduced in this specification describe the embodiments according to the concept of the present disclosure. The embodiments according to the concept of the present disclosure can be practiced in various forms and should not be interpreted as being limited to the embodiments described in the specification.

[0023] Figure 1 is a diagram illustrating an image sensing device 1000 according to an embodiment of the present disclosure.

[0024] Reference Figure 1 , the image sensing device 1000 may operate under the control of the host 3000 .

[0025] The image sensing device 1000 may acquire an image in response to a request from the host 3000. In addition, the image sensing device 1000 may output an image to the host 3000 or a device indicated by the host 3000 in response to a request from the host 3000. Here, the device indicated by the host 3000 may be a memory device storing data or a display device outputting data using a visual method.

[0026] The image sensing device 1000 may be implemented in the form of a package module, a component, etc. In this case, the image sensing device 1000 may be installed in the host 3000. Alternatively, the image sensing device 1000 may be implemented as an electronic device separate from the host 3000.

[0027] The host 3000 can be implemented using any of a variety of electronic devices. For example, the host 3000 can be implemented as a digital camera, a mobile device, a smart phone, a personal computer (PC), a tablet PC, a notebook computer, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a portable multimedia player (PMP), a wearable device, a black box, a robot, an autonomous vehicle, etc.

[0028] The image sensing device 1000 may include an image sensor 100 , an image processor 200 , and a memory 300 .

[0029] The image sensor 100 can acquire an image Img. In detail, when a command for controlling image acquisition is received from the host 3000, the image sensor 100 can acquire an image. The image may include multiple pixel data items (or multiple pixel data). Independent pixel data items may be mapped to each pixel. In other words, the image sensor 100 may acquire an image including multiple pixel data items by acquiring multiple pixel data corresponding to each pixel. The pixel data (or pixel data items) may include information indicating at least one of a position, a color channel, a pixel value, and an exposure value.

[0030] The position of a pixel may represent an arrangement position of the corresponding pixel among a plurality of pixels.

[0031] The color channel of a pixel may represent the color of the light of the corresponding pixel. For example, the color channel may include one of a red channel, a green channel, and a blue channel. The green channel may include a first green channel and a second green channel classified according to the position of the pixel. For example, a pixel with a first green channel may be a pixel in the same row as a pixel with a blue channel, and a pixel with a second green channel may be a pixel in the same row as a pixel with a red channel. In other examples, a pixel with a first green channel may be a pixel in the same column as a pixel with a blue channel, and a pixel with a second green channel may be a pixel in the same column as a pixel with a red channel.

[0032] The pixel value may represent the brightness of the light of the corresponding pixel. In one example, it can be seen that when the pixel value is larger, the light is brighter. The exposure value of the pixel may represent the time for sensing light for the corresponding pixel.

[0033] To this end, the image sensor 100 may be implemented as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0034] The image processor 200 may correct an image. In detail, when an image is acquired by the image sensor 100, the image processor 200 may generate a corrected image Cor_Img in which a gain map is applied to the image. For example, the image processor 200 may generate a corrected image in which a gain value included in the gain map is applied to a pixel. In addition, the image processor 200 may output the corrected image. For example, the image processor 200 may output the corrected image to the host 3000 or a device indicated by the host 3000.

[0035] For this operation, the image sensor 100 may acquire a calibration image Cal_Img. The calibration image is an image used to extract a gain map for correcting an image acquired from the image sensor 100. For example, the calibration image may be an image acquired by the image sensor 100 capturing a white background in an environment with uniform illumination. Due to the structure of the image sensor 100 or the wavelength characteristics of light, distortions such as uneven pixel values ​​for each pixel position or each color channel may appear in the image acquired by the image sensor 100. Since different types of distortion may occur in each image sensor 100, a separate gain map may be required for each image sensor 100. The calibration image may include multiple pixel data items. The calibration image may have the same pixel data array as the image acquired from the corresponding image sensor 100.

[0036] Each gain map may include a plurality of gain values. The gain value may be a parameter for correcting the pixel value of the corresponding pixel data item and may indicate a specific numerical value. The gain map may correspond to the size of the image or calibration image acquired from the image sensor 100. The size of the image or calibration image may be represented by the number of pixel data items arranged in the column direction and the number of pixel data items arranged in the row direction.

[0037] For example, when the number of pixel data items included in the image or calibration image is M×N, the number of multiple gain values ​​included in the gain map may be M×N. Here, M and N are natural numbers. That is, the number of gain values ​​may be the same as the number of pixels. Here, the multiple gain values ​​included in the gain map may correspond one-to-one to the multiple pixel data items included in the image. Each gain value may correspond to the position of each pixel.

[0038] The number of data bits allocated to each gain value may vary with the position of the corresponding pixel. The greater the number of bits allocated to the data, the wider the range of values ​​of the data. For example, when calculating a gain value allocated with a small number of bits, the calculated gain value may be expressed within a small range of values ​​allowed by the small number of bits.

[0039] The image processor 200 according to an embodiment of the present disclosure may use a calibration image to calculate a reference gain value included in a gain map. In detail, the image processor 200 may use a plurality of pixel data items included in the calibration image to calculate a reference gain value corresponding to a vertex of each of a plurality of block regions. The calibration image may include a plurality of block regions. The block region may be defined by a plurality of pixel positions. Therefore, some gain values ​​included in the gain map may be calculated. The image processor 200 may store the reference gain value in the memory 300.

[0040] The gain map may include multiple block areas. The gain map may include multiple gain values. The multiple gain values ​​may include reference gain values ​​and normal gain values. Each of the multiple block areas may be a rectangular area or a square area defined by multiple pixel positions. However, this is only one embodiment, and the multiple block areas may be modified to have various shapes of individual areas.

[0041] Each block area may include a gain value corresponding to a pixel position. Each block area may include a reference gain value. The reference gain value is a gain value corresponding to some of the multiple gain values ​​included in the gain map. Each reference gain value may correspond to a pixel at a preset position. The preset position may be a position corresponding to a vertex of each block area. For example, a block area may be defined by a 4×4 pixel position (or pixel value) ranging from (1,1) to (4,4). The block area may include gain values ​​corresponding to each pixel position from (1,1) to (4,4). In this case, pixel positions (1,1), (4,1), (1,4) and (4,4) may be defined as vertices of the corresponding block area. In the block area, the gain values ​​corresponding to the pixel positions (1,1), (4,1), (1,4) and (4,4) defined as vertices, respectively, may be reference gain values.

[0042] A plurality of block regions according to an embodiment of the present disclosure may include a first block region and a second block region. Here, the distance of the second block region from the center of the image may be shorter than the distance of the first block region from the center of the image (i.e., closer to the center of the image than the first block region), and may have a size larger than that of the first block region. In an embodiment, within an image, as the block region is positioned farther from the center of the image, the size of the block region may become smaller. In an embodiment, within an image, the block region positioned closest to the center of the image may have the largest size, and the block region positioned closest to the edge of the image may have the smallest size.

[0043] In addition, the image processor 200 may calculate the gain value included in each of the plurality of block regions using the reference gain value. Here, the calculated gain value represents the remaining gain values ​​excluding the reference gain value included in each of the plurality of block regions, and hereinafter, for ease of description, each remaining gain value is referred to as a "gain value". Thus, all gain values ​​included in the gain map may be calculated. The image processor 200 may store the gain values ​​included in the gain map in the memory 300.

[0044] The memory 300 may store the reference gain values ​​included in the gain map. In an embodiment of the present disclosure, the memory 300 may store the gain values ​​included in the gain map.

[0045] The memory 300 may be implemented as a nonvolatile memory device. For example, the memory 300 may be implemented as any of various nonvolatile memory devices such as a read-only memory (ROM) that allows only reading of data, a one-time programmable (OTP) memory that allows only writing once, an erasable programmable ROM (EPROM) that can erase stored data and write data, a NAND flash memory, and a NOR flash memory.

[0046] According to the embodiments of the present disclosure, it is possible to provide an image sensing device and a method of operating the image sensing device that can reduce the amount of data to be used for image correction and the calculation load. Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0047] Figure 2 is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0048] Reference Figure 2 , the image sensor 100 may include an optical lens LS, a pixel array 110 , a row decoder 120 , a timing generator 130 , a signal converter 140 , and an output buffer 150 .

[0049] The optical lens LS may refract light that has been reflected from the object and reaches the optical lens LS. The light refracted by the optical lens LS may travel toward the pixel array 110. The optical lens LS may be a lens or a collection of a plurality of lenses arranged on a path along which the light travels. In addition, the optical lens LS may include a collection of microlenses arranged on top of each pixel of the pixel array 110. The object may include at least one of various elements such as an arbitrary object, an animal, a person, and a background that exist outside the image sensor 100.

[0050] The pixel array 110 may include a color filter array and a photoelectric conversion layer. The color filter array may be disposed on the top of the photoelectric conversion layer. The photoelectric conversion layer may be disposed at the bottom of the color filter array. Here, the top and bottom may be defined based on the direction of travel of light, and the light may travel in the direction from the color filter array to the photoelectric conversion layer.

[0051] The color filter array may include a plurality of color filters. For example, each of the plurality of color filters may be one of a red filter, a green filter, and a blue filter. The red filter may transmit light having a wavelength corresponding to red by filtering incident light. The green filter may transmit light having a wavelength corresponding to green by filtering incident light. The blue filter may transmit light having a wavelength corresponding to blue by filtering incident light. However, this is merely an embodiment, and the type of color filter that transmits light of a particular color may be changed in various ways.

[0052] The photoelectric conversion layer may include a plurality of sensing circuits. Each sensing circuit may include a photodiode and a capacitor. The photodiode may generate a current corresponding to light incident thereon through a photoelectric effect. The capacitor may accumulate charge according to the current generated by the photodiode. Here, the accumulated charge amount may correspond to a pixel value indicating brightness.

[0053] The row decoder 120 may select pixels located in a row corresponding to the address in response to the address and the control signal output from the timing generator 130. The pixel array 110 may output a signal corresponding to the amount of accumulated charge from the selected pixel and provide the signal to the signal converter 140.

[0054] The signal converter 140 can obtain multiple pixel data about multiple pixels based on the respective signals output from the pixel array 110. Here, the pixel data may include a pixel value and a pixel color. When describing with one pixel as an example, the signal converter 140 can obtain a pixel value corresponding to the amount of charge accumulated in the capacitor in the photoelectric conversion layer and a color corresponding to the color filter.

[0055] The output buffer 150 may output the image Img or the calibration image Cal_Img. In detail, the output buffer 150 may be implemented using a plurality of buffers storing digital signals output from the signal converter 140. The output buffer 150 may latch and output multiple items of pixel data provided from the signal converter 140 in units of columns. The output buffer 150 may temporarily store multiple items of pixel data output from the signal converter 140, and may sequentially output the multiple items of pixel data under the control of the timing generator 130.

[0056] Figure 3A is a diagram illustrating a pixel array according to an embodiment of the present disclosure.

[0057] Reference Figure 3A According to the embodiment of the present disclosure, the pixel array 110 may be implemented as a Figure 3A Pixel array 310 of the pixel structure shown. Pixel array 310 may include a plurality of pixels. A plurality of pixels may be arranged in a column direction and a row direction. Each of the plurality of pixels may include information about the position where the corresponding pixel is arranged. For example, pixel CE_xy may represent a pixel arranged at the xth position in the column direction and the yth position in the row direction. Here, each of x and y may be a natural number.

[0058] Each of the plurality of pixels may include a microlens ML that refracts light, a color filter CF that transmits light of a specific color, and a sensing circuit PD that detects the intensity of the light. The plurality of pixels may be classified according to the type of the color filter CF. For example, a pixel including a red color filter may be referred to as a pixel having red channels R1 to R4, a pixel including a green color filter may be referred to as a pixel having green channels Gr1 to Gr4 and Gb1 to Gb4, and a pixel including a blue channel color filter may be referred to as a pixel having blue channels B1 to B4.

[0059] The pixel array 310 may include a plurality of pixel groups. The plurality of pixel groups may be arranged in the column direction and the row direction. Each of the plurality of pixel groups may include information about the arrangement position of the corresponding pixel group. For example, the pixel group CG_XY may represent a pixel group arranged at the Xth position in the column direction and the Yth position in the row direction. Here, each of X and Y may be a natural number.

[0060] Each pixel group may include a plurality of pixels arranged in a preset array pattern. That is, the pixel group may represent a unit area that repeats the array pattern of a plurality of pixels. For example, the preset array pattern may be a quaternary Bayer pattern, in which pixels having 2×2 first green channels Gb1 to Gb4, pixels having 2×2 blue channels B1 to B4, pixels having 2×2 red channels R1 to R4, and pixels having 2×2 second green channels Gr1 to Gr4 are arranged in a 4×4 array.

[0061] The above embodiment is only one embodiment, and the preset array pattern may be modified to various patterns, such as a Bayer pattern in which pixels having a 1×1 first green channel, pixels having a 1×1 blue channel, pixels having a 1×1 red channel, and pixels having a 1×1 second green channel are arranged in a 2×2 array.

[0062] Figure 3B is a diagram illustrating an image acquired by an image sensor according to an embodiment of the present disclosure.

[0063] Reference Figure 3B According to an embodiment of the present disclosure, the image Img or the calibration image Cal_Img may be a Figure 3B An image 320 of the pixel structure is shown.

[0064] Image 320 may be acquired by image sensor 100 including pixel array 310 .

[0065] The image 320 may include a plurality of pixel data items. The plurality of pixel data items included in the image 320 may correspond to a plurality of pixels included in the pixel array 310 of the image sensor.

[0066] Each of the plurality of pixel data items may include information about the arrangement position of the corresponding pixel. For example, the pixel data item PX_xy may represent a pixel arranged at the xth position in the column direction and the yth position in the row direction. Here, each of x and y may be a natural number. The pixel data item PX_xy may correspond to a pixel CE_xy having the same position as the pixel data item PX_xy. The pixel data item PX_xy may include information such as a pixel value obtained from the pixel CE_xy.

[0067] The image 320 may include a plurality of pixel data groups. The plurality of pixel data groups may be arranged in the column direction and the row direction. Each of the plurality of pixel data groups may include information about the arrangement position of the corresponding pixel data group. For example, the pixel data group PG_XY may represent a pixel data group arranged at the Xth position in the column direction and the Yth position in the row direction. Here, each of X and Y may be a natural number.

[0068] Each pixel data group may include a plurality of pixel data items arranged in a preset array pattern. That is, the pixel data group may represent a unit area of ​​an array pattern that repeats a plurality of pixel data items. For example, the preset array pattern may be a quaternary Bayer pattern, in which pixel data items having 2×2 first green channels Gb1 to Gb4, pixel data items having 2×2 blue channels B1 to B4, pixel data items having 2×2 red channels R1 to R4, and pixel data items having 2×2 second green channels Gr1 to Gr4 are arranged in a 4×4 array.

[0069] The pixel data items included in the image 320 may be classified according to color channels or channels. For example, the pixel data items included in the image 320 may be classified according to color channels into pixel data items having first green channels Gb1 to Gb4, pixel data items having blue channels B1 to B4, pixel data items having red channels R1 to R4, and pixel data items having second green channels Gr1 to Gr4. In addition, the pixel data items included in the image 320 may be classified according to channels into pixel data items having first channels Gb1 to fourth channels Gb4, pixel data items having fifth channels Gr1 to eighth channels Gr4, pixel data items having ninth channels R1 to twelfth channels R4, and pixel data items having thirteenth channels B1 to sixteenth channels B4.

[0070] In addition, the above embodiment is only one embodiment, and the preset array pattern can be modified to various patterns, such as a Bayer pattern: in which pixel data items having a 1×1 first green channel, pixel data items having a 1×1 blue channel, pixel data items having a 1×1 red channel, and pixel data items having a 1×1 second green channel are arranged in a 2×2 array.

[0071] Figure 4 is a diagram illustrating an image processor according to an embodiment of the present disclosure.

[0072] Reference Figure 4 , the image processor 200 may include an image calibrator 210 and an image corrector 220 .

[0073] The image calibrator 210 may calculate a reference gain value ref_Gain using the calibration image Cal_Img acquired by the image sensor 100 .

[0074] In detail, the image calibrator 210 may select a pixel data item corresponding to a vertex of each block area from among a plurality of pixel data items included in the calibration image Cal_Img acquired by the image sensor 100. Hereinafter, an example in which a reference gain value ref_Gain corresponding to one of the positions of the selected pixel data items is being calculated will be described.

[0075] The image calibrator 210 may calculate an average pixel value of a pixel data item having the same color channel as a pixel data item selected from among the pixel data items included in the region of interest including the selected pixel data item. Here, the same color channel may be one of a first green channel, a second green channel, a red channel, and a blue channel. In an embodiment, the median of the pixel values ​​of the pixel data item may also be used instead of the average pixel value.

[0076] The region of interest may be a region extending to a preset size based on the position of the pixel data items corresponding to the vertices of the block region. In an embodiment, the regions of interest may have the same size. In other embodiments, the regions of interest may have different sizes depending on the position of the pixel data items used as a reference.

[0077] As an example, the same color channel as the selected pixel data item is the first green channel.

[0078] In an embodiment, the average pixel value may be an average value of pixel values ​​of pixel data items having a first green channel among pixel data items included in the region of interest. In other embodiments, the average pixel value may be an average value of pixel values ​​of pixel data items having a first green channel and pixel values ​​of pixel data items having a second green channel among pixel data items included in the region of interest. Even in the case where the color channel that is the same as the color channel of the selected pixel data item is the second green channel, the average value may be calculated in the same manner as described above.

[0079] The image calibrator 210 may calculate a reference gain value ref_Gain corresponding to the selected pixel data item using the pixel value of the selected pixel data item and the calculated average pixel value. For example, the reference gain value ref_Gain may be a ratio of the pixel value of the selected pixel data item to the calculated average pixel value.

[0080] For the remaining pixel data items, among the selected pixel data items, the image calibrator 210 may calculate the reference gain value ref_Gain in this manner. In addition, the image calibrator 210 may store the calculated reference gain value ref_Gain in the memory 300.

[0081] The image calibrator 210 may calculate the gain value Gain using the reference gain value ref_Gain.

[0082] In detail, based on the distance from a point selected from one of the plurality of block areas to the vertex of the one block area and the reference gain value ref_Gain corresponding to the vertex of the one block area, the image calibrator 210 may calculate the gain value Gain corresponding to the selected point. Here, the distance from the vertex to the selected point may be the distance from the pixel data item corresponding to the vertex to the pixel data item corresponding to the selected point. The distance between the pixel data items may correspond to the number of pixel data items existing between the corresponding pixel data items. The distance may be a concept including at least one of the length in the column direction and the length in the row direction.

[0083] In an embodiment, the vertices of one block region may include a first vertex, a second vertex, a third vertex, and a fourth vertex of the one block region.

[0084] In this case, the image calibrator 210 can calculate the gain value Gain corresponding to the selected point based on the distance from the selected point to the first vertex, the distance from the selected point to the second vertex, the distance from the selected point to the third vertex, the distance from the selected point to the fourth vertex, the reference gain value corresponding to the first vertex, the reference gain value corresponding to the second vertex, the reference gain value corresponding to the third vertex, and the reference gain value corresponding to the fourth vertex.

[0085] In an embodiment, the plurality of block regions may include a first block region and a second block region. Here, the distance of the second block region from the center of the image may be shorter than the distance of the first block region from the center of the image, and the second block region may have a size larger than that of the first block region. This utilizes the following phenomenon: the overall distortion is low and the distortion of each region is more uniform in the direction toward the center of the image Img or the calibration image Cal_Img, while the overall distortion is high and the distortion of each region is uneven in the direction toward the edge of the image or the calibration image. When a relatively large-sized block region is set in the direction toward the center of the image Img or the calibration image Cal_Img, and a relatively small-sized block region is set in the direction toward the edge of the image Img or the calibration image Cal_Img, the reference gain value ref_Gain included in the block region closer to the center can be calculated less frequently. The gain value Gain can be calculated more frequently, but when the gain value Gain is calculated using an algorithm that applies a lower calculation load than the reference gain value ref_Gain, the entire calculation load can be reduced.

[0086] In an embodiment, a greater number of bits than the number of bits of the gain value included in the second block area may be allocated to the gain value included in the first block area. Here, the distance of the second block area from the center of the image may be less than the distance of the first block area from the center of the image, and the second block area may have a size larger than the size of the first block area. The number of data bits allocated to each gain value may vary with position or region. The greater the number of bits allocated to the data, the wider the range of values ​​of the data. For example, when calculating a gain value to which a small number of bits are allocated, the calculated gain value may be expressed within a small range of values ​​allowed by the small number of bits.

[0087] When the image Img is acquired by the image sensor 100, the image corrector 220 may generate a corrected image Cor_Img in which the reference gain value ref_Gain and the gain value Gain stored in the memory 300 are applied to a plurality of pixel data items included in the image. Here, the reference gain value ref_Gain and the gain value Gain may be included in a gain map. The reference gain value ref_Gain and the gain value Gain may correspond to the respective pixel data items included in the image.

[0088] The image corrector 220 may output the corrected image Cor_Img. The image corrector 220 may output the corrected image Cor_Img to the host 3000 or a device instructed by the host 3000. For example, the image corrector 220 may output the corrected image Cor_Img to a processor, a display, or a storage in response to a control command received from the host 3000.

[0089] In an embodiment, the image corrector 220 may generate the corrected image Cor_Img using the reference gain value ref_Gain output from the memory 300 and the gain value Gain output from the image calibrator 210 or a volatile memory. That is, the gain value Gain may not be stored in the memory 300 .

[0090] To this end, whenever a preset event occurs, the image calibrator 210 may calculate the gain value Gain using the reference gain value ref_Gain stored in the memory 300. Here, the preset event may include an event that allows the image sensor 100 to be turned on. The image calibrator 210 may transmit the calculated gain value Gain to the image corrector 220. Alternatively, the image calibrator 210 may store the calculated gain value Gain in a volatile memory. The volatile memory may be implemented as a static random access memory (SRAM), a dynamic RAM (DRAM), etc.

[0091] In an embodiment, the image corrector 220 may generate the corrected image Cor_Img using the reference gain value ref_Gain and the gain value Gain output from the memory 300. That is, the gain value Gain may be stored in the memory 300 together with the reference gain value ref_Gain.

[0092] Figure 5 is a diagram illustrating a calibration image according to an embodiment of the present disclosure.

[0093] Reference Figure 5 , the image sensor 100 may acquire an image 500. For example, the image 500 may be a calibration image Cal_Img acquired by the image sensor 100 capturing a white background in an environment with uniform illumination. The image 500 may represent a state before correction is performed. The shadow of each part of the image 500 may represent the brightness of each pixel data item or the pixel value of each pixel data item. It can be seen that the difference in brightness increases in the direction from the center C of the image 500 to the edge of the image 500 away from the center C. It can be seen that the degree of distortion further increases and becomes more uneven in the direction from the center C of the image 500 to the edge of the image 500.

[0094] According to an embodiment of the present disclosure, a plurality of block areas Blk1 and Blk2 may be provided to correct such distortion. The size of each of the plurality of block areas Blk1 and Blk2 may vary with the distance of the corresponding block area from the center C of the image 500 .

[0095] In an embodiment, the size of each of the plurality of block regions Blk1 and Blk2 may decrease as the distance of the corresponding block region from the center C of the image 500 increases. Here, the center C of the image 500 may be a point where the length of the image 500 in the column direction and the length of the image 500 in the row direction are half. For example, the distance between the first block region Blk1 and the center C of the image 500 may be defined as the distance d1 between the center c1 of the first block region Blk1 and the center C of the image 500. In the same manner, the distance between the second block region Blk2 and the center C of the image 500 may be defined as the distance d2 between the center c2 of the second block region Blk2 and the center C of the image 500.

[0096] The above embodiment is only one embodiment, and the reference position as a standard for determining the size of each block area may be changed to various positions, such as the vertex of the image 500, in addition to the center C of the image 500. In addition, the standard for determining the size of each block area may be a ratio of the distance in the column direction to the distance in the row direction. For example, when describing based on the first block area Blk1, the ratio of the distance in the column direction may be a first ratio obtained by dividing the distance b1 in the column direction between the center c1 of the first block area Blk1 and the center C of the image 500 by the length of the image 500 in the column direction. The ratio of the distance in the row direction may be a second ratio obtained by dividing the distance a1 in the row direction between the center c1 of the first block area Blk1 and the center C of the image 500 by the length of the image 500 in the row direction. In addition, when the first ratio and the second ratio are greater than the preset value, the size of the first block area Blk1 may be set to the first size, and when the first ratio and the second ratio are less than the preset value, the size of the first block area Blk1 may be set to the second size greater than the first size.

[0097] Figure 6 is a diagram illustrating a gain graph according to an embodiment of the present disclosure.

[0098] Reference Figure 6 , the gain map 600 may correspond to the size of the image Img acquired from the image sensor 100 or the calibration image Cal_Img.

[0099] The gain map 600 may include a plurality of block regions Blk1, Blk2, and Blk3. In an embodiment, each of the plurality of block regions Blk1, Blk2, and Blk3 may be a square region or a rectangular region. Here, each of the block regions Blk1, Blk2, and Blk3 may be a logically defined region, and each of the block regions Blk1, Blk2, and Blk3 may correspond to an area of ​​the image Img or the calibration image Cal_Img.

[0100] The plurality of block regions Blk1, Blk2, and Blk3 may include a first block region Blk1, a second block region Blk2, and a third block region Blk3. The first block region Blk1, the second block region Blk2, and the third block region Blk3 may be arranged to correspond to the image Img or the calibration image Cal_Img. The first block region Blk1, the second block region Blk2, and the third block region Blk3 may be block regions of different sizes. The arrangement positions of the first block region Blk1, the second block region Blk2, and the third block region Blk3 may change with their sizes. Alternatively, the sizes of the first block region Blk1, the second block region Blk2, and the third block region Blk3 may change with the arrangement positions of the block regions Blk1, Blk2, and Blk3.

[0101] For example, the third block Blk3 having the largest size may be arranged at a position where the distance from the center of the image is shortest. The second block area Blk2 having an intermediate size may be arranged at a position where the distance from the center of the image falls within an intermediate range. The first block area Blk1 having the smallest size may be arranged at a position where the distance from the edge of the image is shortest.

[0102] The image processor 200 may calculate a reference gain value ref_Gain using the calibration image Cal_Img and pixel values ​​of pixel data items included in the block areas Blk1 , Blk2 , and Blk3 .

[0103] In detail, the image processor 200 can select the pixel data items at the preset position from the multiple pixel data items included in the calibration image Cal_Img. Here, the pixel data items at the preset position can be pixel data items corresponding to the vertices of the block areas Blk1, Blk2 and Blk3. The pixel data items at the preset position can be connected by lines in the column direction and the row direction, so that the block areas Blk1, Blk2 and Blk3 can be defined. A reference pixel data group ref_PG_XY can be a pixel data group PG_XY located at a position (X, Y) of a vertex of a block area among the multiple pixel data groups. A pixel data group PG_XY may include pixel data items with different color channels. The reference pixel data group ref_PG_XY may be located at the vertex of at least one block area Blk2 or Blk3. The vertex of a block area may be the vertex of another block area. That is, block areas adjacent to each other may share at least one vertex.

[0104] The image processor 200 can select a region of interest (ROI) corresponding to each vertex of the block areas Blk1, Blk2 and Blk3. Each region of interest (ROI) can be a region having a preset size and having a reference pixel data group ref_PG_XY (i.e., a vertex) as a reference point. Here, the reference point can be the center point of the region of interest ROI. The preset size can be a fixed size. However, this is only one embodiment, and the preset size can be variable according to the position of the reference pixel data group ref_PG_XY (i.e., a vertex). The region of interest (ROI) can include pixel data items located in the corresponding area. The following will refer to Figure 7 The detailed implementation thereof is described.

[0105] Figure 7 is a diagram illustrating any one block region according to an embodiment of the present disclosure.

[0106] Reference Figure 7 , the image processor 200 may select regions of interest ROI1 to ROI4 corresponding to respective vertices Ref_PG1 to Ref_PG4 of one block region BLKi.

[0107] For example, the block area BLKi may have a square or rectangular shape. The length of the block area BLKi in the column direction may be W, and the length thereof in the row direction may be H. Here, W and H may be natural numbers that are equal to or different from each other. The vertices Ref_PG1 to Ref_PG4 of the block area BLKi may include a first vertex Ref_PG1, a second vertex Ref_PG2, a third vertex Ref_PG3, and a fourth vertex Ref_PG4. The first vertex Ref_PG1, the second vertex Ref_PG2, the third vertex Ref_PG3, and the fourth vertex Ref_PG4 may be arranged at a distance W from each other in the column direction and at a distance H in the row direction.

[0108] In this case, the image processor 200 can select a first region of interest ROI1 with a preset size using the position of the first vertex Ref_PG1 as the center point, can select a second region of interest ROI2 with a preset size using the position of the second vertex Ref_PG2 as the center point, can select a third region of interest ROI3 with a preset size using the position of the third vertex Ref_PG3 as the center point, and can select a fourth region of interest ROI4 with a preset size using the position of the fourth vertex Ref_PG4 as the center point.

[0109] The image processor 200 may calculate the reference gain values ​​ref_Gain corresponding to the plurality of vertices Ref_PG1 to Ref_PG4 using the pixel values ​​of the pixel data items included in the respective regions corresponding to the plurality of regions of interest ROI1 to ROI4 in the calibration image Cal_Img. That is, the reference gain value ref_Gain of each of the plurality of vertices Ref_PG1 to Ref_PG4 may be calculated using the pixel values ​​included in each of the plurality of vertices and the adjacent pixel values.

[0110] For example, the image processor 200 may calculate a reference gain value corresponding to the first vertex Ref_PG1 of the block area BLKi using the pixel values ​​of the pixel data items included in the area corresponding to the first region of interest ROI1 in the calibration image Cal_Img. The image processor 200 may calculate a reference gain value corresponding to the second vertex Ref_PG2 of the block area BLKi using the pixel values ​​of the pixel data items included in the area corresponding to the second region of interest ROI2 in the calibration image Cal_Img. The image processor 200 may calculate a reference gain value corresponding to the third vertex Ref_PG3 of the block area BLKi using the pixel values ​​of the pixel data items included in the area corresponding to the third region of interest ROI3 in the calibration image Cal_Img. The image processor 200 may calculate a reference gain value corresponding to the fourth vertex Ref_PG4 of the block area BLKi using the pixel values ​​of the pixel data items included in the area corresponding to the fourth region of interest ROI4 in the calibration image Cal_Img.

[0111] The image processor 200 may calculate the remaining gain values ​​Gain included in the block area BLKi using the reference gain values ​​ref_Gain corresponding to the plurality of vertices Ref_PG1 to Ref_PG4. Hereinafter, reference will be made to the following equations (1-1) to (3) and FIG. 8A to FIG. 8C A detailed implementation of calculating the reference gain value is described.

[0112] FIG. 8A to FIG. 8C is a diagram illustrating a method of calculating a reference gain value according to an embodiment of the present disclosure.

[0113] The image processor 200 may calculate the reference gain value ref_Gain_xy using equations (1-1) to (3).

[0114]

[0115] In equations (1-1) to (3), the pixel value ref_PV_xy represents the pixel value of the selected pixel data item. The selected pixel data item represents one pixel data item selected from among pixel data items included in the pixel data group PG_XY having the same arrangement position as the arrangement position of the vertex.

[0116] A case is described in which the selected pixel data item is one of the pixel data items having the first green channel Gb1 to Gb4.

[0117] In an embodiment, as shown in formula (1-1), the image processor 200 may calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data item and the average pixel value ROI_AvgPV_G of the green channels Gb1 to Gb4 and Gr1 to Gr4. Here, the average pixel value ROI_AvgPV_G of the green channels Gb1 to Gb4 and Gr1 to Gr4 may be the average value of the pixel values ​​of the pixel data items having the first green channels Gb1 to Gb4 and the pixel data items having the second green channels Gr1 to Gr4 among the plurality of pixel data items included in the corresponding region of interest (ROI).

[0118] In an embodiment, as shown in formula (1-2), the image processor 200 may calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data item and the average pixel value ROI_AvgPV_Gb of the first green channels Gb1 to Gb4. Here, the average pixel value ROI_AvgPV_Gb of the first green channels Gb1 to Gb4 may be an average value of the pixel values ​​of the pixel data items having the first green channels Gb1 to Gb4 among the plurality of pixel data items included in the region of interest (ROI).

[0119] A case is described in which the selected pixel data item is one of the pixel data items having the second green channel Gr1 to Gr4.

[0120] In an embodiment, as shown in equation (1-1), the image processor 200 may calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data item and the average pixel value ROI_AvgPV_G of the green channels Gb1 to Gb4 and Gr1 to Gr4.

[0121] In an embodiment, as shown in formula (1-3), the image processor 200 may calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data item and the average pixel value ROI_AvgPV_Gr of the second green channels Gr1 to Gr4. Here, the average pixel value ROI_AvgPV_Gr of the second green channels Gr1 to Gr4 may be the average value of the pixel values ​​of the pixel data items having the second green channels Gr1 to Gr4 among the plurality of pixel data items included in the region of interest (ROI).

[0122] A case is described in which the selected pixel data item is one of the pixel data items having red channels R1 to R4. In this case, as shown in equation (2), the image processor 200 may calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data item and the average pixel value ROI_AvgPV_R of the red channels R1 to R4. The average pixel value ROI_AvgPV_R of the red channels R1 to R4 may be the average value of the pixel values ​​of the pixel data items having red channels R1 to R4 among the plurality of pixel data items included in the region of interest (ROI).

[0123] A case is described in which the selected pixel data item is one of the pixel data items having blue channels B1 to B4. In this case, as shown in equation (3), the image processor 200 may calculate the reference gain value ref_Gain_xy using the pixel value ref_PV_xy of the selected pixel data item and the average pixel value ROI_AvgPV_B of the blue channels B1 to B4. The average pixel value ROI_AvgPV_B of the blue channels B1 to B4 may be the average value of the pixel values ​​of the pixel data items having blue channels B1 to B4 among the plurality of pixel data items included in the region of interest (ROI).

[0124] In the following, reference will be made to Fig. 8A and Figure 8B Describes how to calculate a reference gain value.

[0125] Reference Fig. 8A and Figure 8B , the image processor 200 may select one pixel data item from among the pixel data items in the pixel data group PG_XY having the same arrangement positions as the arrangement positions of the corresponding vertices of the block area Blki in the calibration image 810 or 820. Here, the pixel data item having the pixel value ref_PX_xy of the first channel Gb1 set at (x, y) is selected.

[0126] The image processor 200 may select a region of interest (ROI) obtained by extending the pixel data group PG_XY as a central area to a preset size in the calibration image 810 or 820 .

[0127] In an embodiment, if Fig. 8A As shown, the image processor 200 can select a pixel data item ROI_Avg_Gb having a first green channel Gb1 to Gb4 from among a plurality of pixel data items included in a region of interest (ROI) in the image 810, the first green channel Gb1 to Gb4 being the same color channel as the color channel of the selected pixel data item having the pixel value ref_PX_xy. The image processor 200 can calculate an average pixel value ROI_AvgPV_Gb as the average value of the pixel values ​​of the selected pixel data item ROI_Avg_Gb. As shown in formula (1-2), the image processor 200 can calculate a reference gain value ref_Gain_xy of the position (x, y) using the pixel value ref_PV_xy of the pixel data item selected at the position (x, y) and the average pixel value ROI_AvgPV_Gb. In this way, reference gain values ​​of other positions can be calculated. In an embodiment, the pixel value ref_PV_xy in equation (1-2) may be replaced by the average value TOT_Avg_Gb1 of the Gb1 pixel values ​​included in the ROI.

[0128] In an embodiment, if Figure 8B As shown, the image processor 200 can select a pixel data item ROI_Avg_G having a first green channel Gb1 to Gb4 and a second green channel Gr1 to Gr4 from among a plurality of pixel data items included in a region of interest (ROI) in the image 820, the first green channel Gb1 to Gb4 and the second green channel Gr1 to Gr4 being a color channel of the same type as the color channel of the selected pixel data item having the pixel value ref_PX_xy. The image processor 200 can calculate an average pixel value ROI_AvgPV_G which is an average value of the pixel values ​​of the selected pixel data item ROI_Avg_G. As shown in formula (1-1), the image processor 200 can calculate a reference gain value ref_Gain_xy of the position (x, y) using the pixel value ref_PV_xy of the pixel data item selected at the position (x, y) and the average pixel value ROI_AvgPV_G. In this way, reference gain values ​​of other positions can be calculated. In an embodiment, the pixel value ref_PV_xy in equation (1-1) may be replaced by the average value TOT_Avg_Gb1 of the Gb1 pixel values ​​included in the ROI.

[0129] Return to reference Figure 6, the first block Blk1 having the smallest size among the plurality of blocks Blk1, Blk2, and Blk3 may be arranged in the edge region 630. According to an embodiment of the present disclosure, in the case where the first block Blk1 is arranged in the edge region 630, the reference gain value of the first block Blk1 may be calculated by a mirroring method. This will be referred to below. Figure 8C Give a description.

[0130] Reference Figure 8C , in the edge region 830, the first block Blk1 having the smallest size may be arranged, and the plurality of vertices 850 and 860 of the first block Blk1 may include a first vertex 850 and a second vertex 860, the first vertex 850 being closer to the center of the image than the second vertex 860 (i.e., closer to the center). Among all the vertices of all the block regions, the second vertex 860 may be closest to the first vertex 850. That is, the second vertex 860 may be a vertex disposed at the outermost position of the edge of the image among all the vertices.

[0131] The image processor 200 may calculate a first reference gain value corresponding to the first vertex 850 closer to the center. In detail, the image processor 200 may select a reference pixel data group ref_PG_XY corresponding to the position (X, Y) of the first vertex 850 from the calibration image Cal_Img, and may select the region of interest 840 based on the reference pixel data group ref_PG_XY. The image processor 200 may calculate a first reference gain value corresponding to the position (X, Y) using pixel data items included in the region of interest 840 and pixel values ​​ref_PV_xy of pixel data items included in the reference pixel data group ref_PG_XY.

[0132] In addition, the image processor 200 may copy the first reference gain value corresponding to the first vertex 850 to the second reference gain value corresponding to the second vertex 860. That is, the second reference gain value of the second vertex 860 may be the same value as the first reference gain value of the first vertex 850.

[0133] Fig.9A and Fig. 9B is a diagram illustrating a method of calculating a gain value according to an embodiment of the present disclosure.

[0134] Reference Fig.9A and Fig. 9B , a block area Blki can be defined by multiple vertices. A block area Blki can include points with specific positions. A point can correspond to a pixel with the same position.

[0135] The image processor 200 may use reference gain values ​​ref_PX1 to ref_PX4 corresponding to a plurality of vertices defining a block region Blki to calculate gain values ​​corresponding to a plurality of points included in a block region Blki, respectively. In this case, the image processor 200 may use reference gain values ​​ref_PX1 to ref_PX4 corresponding to a plurality of vertices of the same channel to calculate the gain value of the same channel. Here, a method of calculating a gain value Gain_P_xy corresponding to a point P_xy among a plurality of points included in a block region Blki is described. Here, the channel is the first channel Gb1.

[0136] The image processor 200 may obtain the distance W in the column direction and the distance H in the row direction between the plurality of vertices defining a block area Blki. For example, the image processor 200 may obtain the distance W in the column direction and the distance H in the row direction based on the difference between the positions of the plurality of vertices.

[0137] The image processor 200 can obtain the distances x1 and x2 in the column direction and the distances y1 and y2 in the row direction between each vertex defining a block area Blki and a selected point P_xy. For example, the image processor 200 can obtain the distances x1 and x2 in the column direction and the distances y1 and y2 in the row direction based on the difference between the positions of each vertex and a selected point P_xy.

[0138] The image processor 200 may be based on Fig.9A The formula shown A gain value Gain_P_xy corresponding to one point P_xy is calculated using reference gain values ​​ref_PX1 to ref_PX4 respectively corresponding to a plurality of vertices, distances x1 , x2 , and W in the column direction, and distances y1 , y2 , and H in the row direction.

[0139] Fig.10 is a diagram illustrating a corrected image according to an embodiment of the present disclosure.

[0140] Reference Fig.10 The image processor 200 may generate a corrected image Cor_Img by applying a gain map to the image Img acquired by the image sensor 100 .

[0141] In detail, the image processor 200 may multiply the pixel value I_xy of the pixel data item at the position (x, y) selected from among the plurality of pixel data items included in the image Img by a gain value Gain_xy corresponding to the selected position (x, y) among the plurality of reference gain values ​​and the plurality of gain values ​​included in the gain map. In this case, the image processor 200 may generate a corrected image Cor_Img having a result value O_xy obtained by multiplying the pixel value I_xy by the corresponding gain value Gain_xy for each position as a pixel value.

[0142] That is, the pixel values ​​of the multiple pixel data items included in the corrected image Cor_Img can be the result values ​​obtained by multiplying the pixel values ​​of the multiple pixel data items included in the image Img by the multiple reference gain values ​​and multiple gain values ​​included in the gain map for each corresponding position.

[0143] Fig.11 is a flowchart illustrating a method of operating an image sensing device according to an embodiment of the present disclosure.

[0144] Reference Fig.11 According to the method of operating the image sensing device 1000 according to an embodiment of the present disclosure, in operation S1110 , a calibration image Cal_Img is acquired through the image sensor 100 .

[0145] In step S1120, a reference gain value corresponding to a vertex of each of the plurality of block regions Blk1, Blk2, and Blk3 in the calibration image Cal_Img may be calculated. That is, a reference gain value corresponding to a vertex defined by the plurality of block regions Blk1, Blk2, and Blk3 included in a gain map corresponding to a size of the calibration image Cal_Img may be calculated. In an embodiment, each of the plurality of block regions Blk1, Blk2, and Blk3 may be a rectangular region or a square region.

[0146] Here, the plurality of block regions Blk1 , Blk2 , and Blk3 may include a first block region Blk1 and a second block region Blk2 that is closer to the center of the calibration image Cal_Img than the first block region Blk1 and has a size larger than that of the first block region Blk1 .

[0147] In an embodiment, the operation of calculating the reference gain value may include: an operation of selecting pixel data items corresponding to vertices of multiple block areas Blk1, Blk2 and Blk3 from multiple pixel data items included in the calibration image Cal_Img acquired by the image sensor 100, an operation of calculating the average pixel value of pixel data items having a selected color channel among pixel data items included in a region of interest (ROI) including one of the selected pixel data items, and an operation of calculating one of the reference gain values ​​using the pixel value of the selected pixel data item and the average pixel value.

[0148] In an embodiment, the same color channel may be a color channel corresponding to one of a first green channel, a second green channel, a red channel, and a blue channel.

[0149] In an embodiment, the same color channel may be a color channel corresponding to one of the first green channel and the second green channel. In this case, the average pixel value may include an average value of the pixel values ​​of the pixel data items having the first green channel and the pixel data items having the second green channel among the pixel data items included in the region of interest.

[0150] Also, in operation S1130, gain values ​​corresponding to respective points included in the plurality of block regions Blk1, Blk2, and Blk3 may be calculated using reference gain values.

[0151] In an embodiment, the operation of calculating the gain value may include: based on the distance from a point selected from a block area Blki among multiple block areas Blk1, Blk2 and Blk3 to the vertex of the block area Blki and the reference gain value corresponding to the vertex of the block area Blki, the operation of calculating the gain value corresponding to the selected point.

[0152] In an embodiment, the vertices of a block area Blki may include a first vertex, a second vertex, a third vertex, and a fourth vertex of a block area Blki. In this case, the operation of calculating the gain value corresponding to the selected point may be configured as: based on the distance from the selected point to the first vertex, the distance from the selected point to the second vertex, the distance from the selected point to the third vertex, the distance from the selected point to the fourth vertex, the reference gain value corresponding to the first vertex, the reference gain value corresponding to the second vertex, the reference gain value corresponding to the third vertex, and the reference gain value corresponding to the fourth vertex, the gain value corresponding to the selected point is calculated.

[0153] Furthermore, in operation S1140 , the reference gain value and the gain value may be stored in the memory 300 .

[0154] In an embodiment, a greater number of bits than the number of bits of the gain values ​​included in the second block area Blk2 may be allocated to the gain values ​​included in the first block area Blk1 .

[0155] In an embodiment, the operating method of the image sensing device 1000 may calculate a reference gain value of a reference position within each block region from a calibration image. The operating method may generate a gain map by calculating the remaining gain values ​​of the remaining positions within the block region based on the reference gain value of the block region, the gain map including the reference gain value and the remaining gain values ​​corresponding to the respective pixel data items within the block region. The operating method may generate a corrected image by applying the gain map to an input image. Here, the calibration image and the input image may have geometrically identical pixel data arrays. The block region may be divided into two or more groups located in respective layer regions, which are radially layered with the center of the calibration image as a reference. The block regions located in one of the layer regions may have the same size. The block regions located in the layer region relatively close to the center may have a larger size than the block regions located in the layer region relatively far from the center.

[0156] In an embodiment, the reference position may be a vertex of the block region.

[0157] Fig.12 is a flowchart illustrating a method of operating an image sensing device according to an embodiment of the present disclosure.

[0158] Reference Fig.12 The method of operating the image sensing device 1000 according to an embodiment of the present disclosure may acquire an image Img through the image sensor 100 at operation S1210. When a capture command is received from the host 3000, the image sensing device 1000 may acquire the image Img through the image sensor 100.

[0159] In addition, in operation S1220, a correction image Cor_Img may be generated in which the reference gain value and the gain value stored in the memory 300 are applied to a plurality of pixel data items. The reference gain value and the gain value stored in the memory 300 may be included in a gain map. The gain map may correspond to the size of the calibration image Cal_Img or the image Img. The reference gain value and the gain value included in the gain map may correspond to a plurality of pixels.

[0160] In addition, the correction image Cor_Img may be output at operation S1230. The image sensing device 1000 may output the correction image Cor_Img to the host 3000. Alternatively, the image sensing device 1000 may output the correction image Cor_Img to a device indicated by the host 3000. For example, the correction image Cor_Img may be output to an external device of the image sensing device 1000 (such as a storage device, a processor, or a display).

[0161] Fig.13 is a diagram illustrating a computing system including an image sensing device according to an embodiment of the present disclosure.

[0162] Reference Fig.13 , the computing system 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output device 2050, and a display device 2060. Fig.13 Not shown, the computing system 2000 may further include a port through which the computing system 2000 can communicate with a storage device 2030 , a memory device 2040 , an input / output (I / O) device 2050 , a display device 2060 , etc., or can communicate with an external device.

[0163] The image sensor 2010 may acquire an image or a calibration image. The image sensor 2010 may store a gain map. The image sensor 2010 may generate a correction image Cor_Img in which the gain map is applied to the acquired image. The image sensor 2010 may be connected to the processor 2020 through an address bus, a control bus, and a data bus or an additional communication link different from the bus, and then may communicate with the processor 2020. Here, the description of the image sensing device 1000 described above may be applied to the image sensor 2010.

[0164] The image sensor 2010 may be implemented in various types of packages. For example, at least some components of the image sensor 2010 may be implemented using any of the following packages: package stack (PoP), ball grid array (BGA), chip scale package (CSP), plastic lead chip carrier (PLCC), plastic dual in-line package (PDIP), wafer in waffle package, wafer in wafer form, chip on board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline integrated circuit (SOIC), shrink small outline package (SSOP), thin small outline package (TSOP), system-in-package (SIP), multi-chip package (MCP), wafer-level manufacturing package (WFP), or wafer-level processing stacked package (WSP). Depending on the embodiment, the image sensor 2010 and the processor 2020 may be integrated into a single chip, or may be integrated into different chips respectively.

[0165] The processor 2020 may control the overall operation of the computing system 2000. The processor 2020 may control the display device 2060 to display the correction image Cor_Img. The processor 2020 may store the correction image Cor_Img in the storage device 2030. Here, the above description of the host 3000 may be applied to the processor 2020.

[0166] The processor 2020 may perform specific calculations or tasks. According to an embodiment of the present disclosure, the processor 2020 may include at least one of a central processing unit (CPU), an application processing unit (APU), and a graphics processing unit (GPU).

[0167] The processor 2020 may be connected to the storage device 2030, the memory device 2040, and the input / output device 2050 through an address bus, a control bus, and a data bus, and then may communicate with these devices. According to an embodiment of the present disclosure, the processor 2020 may also be connected to an expansion bus such as a peripheral component interconnect (PCI) bus.

[0168] The storage device 2030 may store data such as the correction image Cor_Img. Here, the data stored in the storage device 2030 may be retained not only when the computing system 2000 is driven, but also when the computing system 2000 is not driven. For example, the storage device 2030 may be implemented as at least one of all types of nonvolatile memory devices such as a flash memory device, a solid state drive (SSD), a hard disk drive (HDD), and an optical disk.

[0169] The memory device 2040 may store data such as the correction image Cor_Img. Here, the data stored in the memory device 2040 may be retained only when the computing system 2000 is driven. Alternatively, the data stored in the memory device 2040 may be retained when the computing system 2000 is driven or not driven. For example, the memory device 2040 may include a volatile memory device such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), and a non-volatile memory device such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory device.

[0170] The input / output device 2050 may include an input device and an output device. The input device may be a device capable of inputting a user command through interaction, and may be implemented as, for example, a keyboard, a keypad, a mouse, or a microphone. The output device may be a device capable of outputting data, and may be implemented as, for example, a printer or a speaker.

[0171] The display device 2060 may be a device that visually outputs data (such as a corrected image). For this operation, the display device 2060 may be implemented as any of the following various types of display devices: a liquid crystal display (LCD) that uses a separate backlight unit (e.g., a light emitting diode (LED), etc.) as a light source and controls the transmittance (e.g., brightness or intensity of light) of light emitted from the backlight unit through the liquid crystal by controlling the molecular arrangement of the liquid crystal; and a display device that uses a self-luminous element (e.g., a micro LED with a size of 100um to 200um, a micro LED with a size of 100um or less, an organic LED (OLED), a quantum dot LED (QLED), etc.) as a light source without a separate backlight unit or liquid crystal.

[0172] The display device 2060 may include a plurality of pixels. The plurality of pixels of the display device 2060 may correspond in position to the plurality of pixel data items of the correction image Cor_Img. The plurality of pixels of the display device 2060 may emit light having a brightness value corresponding to each pixel value of the plurality of pixel data items of the correction image Cor_Img, so that an image may be displayed. The display device 2060 may include a plurality of drive circuits corresponding to the plurality of pixels. Here, each drive circuit may be implemented in a form such as an amorphous silicon (a-Si) thin film transistor (TFT), a low temperature polycrystalline silicon (LTPS) TFT, or an organic TFT (OTFT).

[0173] In an embodiment, the display device 2060 may be implemented as a flexible display having a characteristic in which the shape of the display device 2060 is bendable and restorable. In an embodiment, the display device 2060 may be implemented as a transparent display having a light-transmitting characteristic. In an embodiment, the display device 2060 may be implemented as a touch display after being combined with a touch sensor for recognizing a position touched by a user.

[0174] The present disclosure can provide an image sensing device and a method of operating the image sensing device that can reduce the amount of data to be used for image correction and the calculation load.

[0175] Although the present disclosure has been illustrated and described with reference to specific embodiments and the accompanying drawings, the disclosed embodiments are not intended to be limiting. In addition, it should be noted that the present disclosure may be implemented in various ways by replacement, change and modification, as will be appreciated by those skilled in the art based on the present disclosure, without departing from the spirit and / or scope of the present disclosure and the appended claims. In addition, the embodiments may be combined to form additional embodiments.

[0176] CROSS-REFERENCE TO RELATED APPLICATIONS

[0177] This application claims priority to Korean Patent Application No. 10-2021-0059457, filed on May 7, 2021, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An image sensing device, comprising: an image sensor that acquires an image including a plurality of pixel values; a memory storing a reference gain value corresponding to each of a plurality of block regions included in the image; as well as an image processor that calculates a gain value included in each of the plurality of block regions using the reference gain value and outputs a corrected image in which the reference gain value and the gain value are applied to the plurality of pixel values, The plurality of block regions include a first block region and a second block region, the second block region being closer to the center of the image than the first block region and having a size larger than that of the first block region, Each of the third block areas which is farthest from the center of the image and has the smallest size among the plurality of block areas includes a first vertex and a second vertex which is closest to the first vertex and farther from the center of the image than the first vertex, and The first reference gain value corresponding to the first vertex is copied to obtain the second reference gain value corresponding to the second vertex.

2. The image sensing device according to claim 1, wherein: A greater number of bits than the number of bits allocated to each gain value included in the second block area is allocated to each gain value included in the first block area.

3. The image sensing device according to claim 1, wherein: The image processor includes an image calibrator, which: selecting a pixel value corresponding to the vertex of each of the plurality of block areas from among a plurality of pixel values ​​included in a calibration image acquired by the image sensor, calculating an average pixel value of pixel values ​​having the same color channel as that of one of the selected pixel values ​​among pixel values ​​included in the region of interest including each selected pixel value, and One of the reference gain values ​​is calculated using the pixel value of each of the selected pixel values ​​and the average pixel value.

4. The image sensing device according to claim 3, wherein: The same color channel is one of a first green channel, a second green channel, a red channel, and a blue channel.

5. The image sensing device according to claim 3, wherein: The same color channel is one of a first green channel and a second green channel, and The average pixel value includes an average value of pixel values ​​having the first green channel and pixel values ​​having the second green channel among pixel values ​​included in the region of interest.

6. The image sensing device according to claim 1, wherein: The image processor includes an image calibrator that calculates a gain value corresponding to a selected point based on a distance from a point selected from one of the plurality of block areas to a vertex of the one block area and a reference gain value corresponding to the vertex of the one block area.

7. The image sensing device according to claim 6, wherein: The vertices of the one block area include the first vertex, the second vertex, the third vertex, and the fourth vertex of the one block area, and The image calibrator calculates a gain value corresponding to the selected point based on the distance from the selected point to the first vertex, the distance from the selected point to the second vertex, the distance from the selected point to the third vertex, the distance from the selected point to the fourth vertex, the reference gain value corresponding to the first vertex, the reference gain value corresponding to the second vertex, the reference gain value corresponding to the third vertex, and the reference gain value corresponding to the fourth vertex.

8. The image sensing device according to claim 1, wherein: The image processor also stores the reference gain value and a gain value calculated using the reference gain value in the memory.

9. The image sensing device according to claim 8, wherein: The image processor includes an image corrector that generates the corrected image in which the reference gain value and the gain value stored in the memory are applied to the plurality of pixel values ​​included in the image when the image is acquired by the image sensor.

10. The image sensing device according to claim 1, wherein: Each of the plurality of block areas is a rectangular area or a square area.

11. A method of operating an image sensing device, the method comprising the steps of: calculating a reference gain value corresponding to each of the vertices of a plurality of block regions included in the calibration image acquired by the image sensing device; calculating a gain value included in each of the plurality of block regions using the reference gain value; storing the reference gain value and the gain value in a memory; as well as When an image is acquired by the image sensing device, outputting a corrected image in which the reference gain value and the gain value stored in the memory are applied to a plurality of pixel values ​​within the image, The plurality of block regions include a first block region and a second block region, the second block region being shorter than the first block region from the center of the calibration image and having a size larger than that of the first block region, Each of the third block areas which is farthest from the center of the image and has the smallest size among the plurality of block areas includes a first vertex and a second vertex which is closest to the first vertex and farther from the center of the image than the first vertex, and The first reference gain value corresponding to the first vertex is copied to obtain the second reference gain value corresponding to the second vertex.

12. The method according to claim 11, wherein: A greater number of bits than the number of bits allocated to each gain value included in the second block area is allocated to each gain value included in the first block area.

13. The method according to claim 11, wherein: The step of calculating the reference gain value comprises: selecting a pixel value corresponding to the vertex of each of the plurality of block regions from among a plurality of pixel values ​​included in the calibration image acquired by the image sensing device, calculating an average pixel value of pixel values ​​having the same color channel as that of one of the selected pixel values ​​among pixel values ​​included in the region of interest including each selected pixel value, and One of the reference gain values ​​is calculated using the pixel value of each of the selected pixel values ​​and the average pixel value.

14. The method according to claim 13, wherein: The same color channel is one of a first green channel, a second green channel, a red channel, and a blue channel.

15. The method according to claim 13, wherein: The same color channel is one of a first green channel and a second green channel, and The average pixel value includes an average value of pixel values ​​having the first green channel and pixel values ​​having the second green channel among pixel values ​​included in the region of interest.

16. The method according to claim 11, wherein: The step of calculating the gain value includes the following steps: calculating a gain value corresponding to the selected point based on a distance from a point selected from one of the multiple block areas to a vertex of the one block area and a reference gain value corresponding to the vertex of the one block area.

17. The method according to claim 16, wherein: The vertices of the one block area include the first vertex, the second vertex, the third vertex, and the fourth vertex of the one block area, and The step of calculating the gain value corresponding to the selected point comprises the following steps: A gain value corresponding to the selected point is calculated based on the distance from the selected point to the first vertex, the distance from the selected point to the second vertex, the distance from the selected point to the third vertex, the distance from the selected point to the fourth vertex, the reference gain value corresponding to the first vertex, the reference gain value corresponding to the second vertex, the reference gain value corresponding to the third vertex, and the reference gain value corresponding to the fourth vertex.

18. The method according to claim 11, wherein: Each of the plurality of block areas is a rectangular area or a square area.

Images